Fix Gunicorn’s TimeoutError: [Errno 110] Connection timed out in sock.sendall()

If gunicorn is logging TimeoutError: [Errno 110] Connection timed out from sock.sendall(), your view is almost certainly fine. The worker finished the response and then got stuck handing it over. Raising --timeout — the usual first move — makes the problem last longer instead of fixing it.

The log signature

It looks like this, and the last two frames are the whole story:

[ERROR] Error handling request /reports/annual/
Traceback (most recent call last):
  File ".../gunicorn/workers/sync.py", line 137, in handle
    self.handle_request(listener, req, client, addr)
  File ".../gunicorn/workers/sync.py", line 189, in handle_request
    resp.write(item)
  File ".../gunicorn/http/wsgi.py", line 335, in write
    util.write(self.sock, arg)
  File ".../gunicorn/util.py", line 296, in write
    sock.sendall(data)
TimeoutError: [Errno 110] Connection timed out

(Line numbers move between gunicorn versions; the frames don’t. A streaming response is the one exception — there write() is called per chunk while your generator is still producing. Everything below assumes the ordinary buffered case.)

Note where it is in the stack. There is no application code in that traceback at all — no Django, no view, no ORM. By the time resp.write() runs, your view has already returned. The request was served. What failed was writing the bytes to the socket.

What errno 110 actually means on a write

Errno 110 is ETIMEDOUT. On a write, that is not “the other side was slow.” It means the kernel put your data on the wire, waited for a TCP acknowledgement, retransmitted when none came, waited longer, retransmitted again — and eventually gave up.

Somebody stopped reading and didn’t say so. A client on a dead mobile connection, a proxy that dropped the flow, a load balancer that reaped an idle connection without sending a RST.

It is worth separating the three socket errors that show up in this same position, because they mean different things:

• [Errno 32] Broken pipe (EPIPE) — the peer closed properly. You wrote to a socket it had already shut down. Normal background noise; users hit stop.
• [Errno 104] Connection reset by peer (ECONNRESET) — the peer sent a RST. Abrupt, but it told you.
• [Errno 110] Connection timed out (ETIMEDOUT) — the peer said nothing at all and the retransmission timer expired.

Only the third one takes real time to happen, and that gives you a free diagnostic.

The give-away: check how long the request took

Linux retransmits an established-connection write according to tcp_retries2:

cat /proc/sys/net/ipv4/tcp_retries2

The default is 15. The kernel documentation is unusually specific about what that buys you: a hypothetical timeout of 924.6 seconds — about 15 minutes — and it notes this is a lower bound on the effective timeout.

So if your access log shows the request occupying a worker for fifteen minutes, stop looking for a slow query. Nothing in your code is slow enough to be interesting at that timescale, and no ORM call takes a quarter of an hour and then throws a socket error. That duration is the retransmission timer. It is a fingerprint.

Why it only happens on your biggest pages

This is the part that sends people hunting for a bug in one specific view, and there isn’t one.

When a response is small, sendall() copies it into the kernel’s socket send buffer and returns immediately. The worker moves on to the next request, cheerfully unaware that the client vanished. The failure happens and nobody notices.

When a response is large, it doesn’t fit. sendall() fills the send buffer, then blocks waiting for the peer’s receive window to open so it can write the rest. Now the worker is parked inside a blocking write, holding a whole process, for as long as the kernel keeps retrying.

The endpoint that returns 8 MB of uncompressed HTML isn’t broken. It is just the only one whose response is big enough to make the write block long enough for you to see it.

Why raising --timeout is the wrong instinct

Gunicorn’s --timeout is a worker liveness timeout, not a network one. The arbiter kills workers that haven’t checked in recently, on the assumption they’ve hung.

Here, the arbiter is doing exactly the right thing: it notices a worker stuck for 30 seconds and recycles it. The mistake is reading that kill as the fault. Set --timeout 3600 and you have not repaired a single socket — you have just given each doomed write permission to hold a worker for an hour. With the sync worker’s one-request-at-a-time model and a handful of workers, a modest trickle of these will consume your entire pool. Then healthy requests start queueing, and the proxy in front starts returning 502s for endpoints that work perfectly.

Which raises a question the arithmetic answers. The default --timeout is 30 seconds; the kernel’s write timer is around 924. The arbiter wins that race every single time — so with stock settings you never reach ETIMEDOUT at all. You get [CRITICAL] WORKER TIMEOUT instead.

So if [Errno 110] is in your logs, one of two things is already true. Either you are running a threaded worker, where a thread blocked in sendall() doesn’t stop the worker heartbeating, so nothing interrupts the write — or, far more often, you already raised --timeout to silence the WORKER TIMEOUT messages, and this is what was underneath them. The traceback is frequently a symptom of the first fix people reach for.

That is how this failure looks like a capacity problem when it is really a plumbing problem.

None of which means slow requests don’t exist. If a view genuinely needs ninety seconds of work, the answer is to get that work off the request path — a task queue, a job runner, anything the client isn’t sitting and waiting on — rather than blinding the arbiter so it stops telling you about it.

The fix: never let gunicorn write to the internet

Gunicorn’s own documentation is blunt about this — it expects a buffering reverse proxy in front of it, and this is the failure it’s protecting you from. A proxy accepts the whole response from gunicorn at local-network speed, frees the worker immediately, and then deals with the slow client on its own time.

With nginx, buffering is on by default, but the defaults are small enough that a large response still backs up into the worker. Raise them:

# /etc/nginx/sites-available/MY_PROJECT
location / {
    proxy_pass http://unix:/var/www/MY_PROJECT/MY_PROJECT.sock;
 
    proxy_buffering on;
    proxy_buffers 16 64k;
    proxy_buffer_size 64k;
    proxy_busy_buffers_size 128k;
 
    # Already the default - but confirm nobody has set it to 0, which disables
    # the disk spill and pushes back-pressure straight onto gunicorn.
    proxy_max_temp_file_size 1024m;
}

Then stop sending so many bytes. Compression is the single highest-leverage line here, because it attacks the size directly:

# /etc/nginx/nginx.conf
gzip on;
# text/html is always compressed - listing it here does nothing.
gzip_types text/css application/json application/javascript text/xml;
gzip_min_length 1024;

Large HTML pages are mostly repeated markup and typically compress by 70–90%. An 8 MB page usually lands around a megabyte, which fits comfortably in the buffers above and stops blocking at all.

If you are on a managed platform whose front end you don’t control — an App Service, an ALB, a CDN in front of the origin — the same principle applies, you just configure it elsewhere. Confirm the front end buffers responses, and confirm compression is actually switched on rather than merely available.

Sometimes there is no front end you can configure at all. Then compress in the application instead, with GZipMiddleware placed first so it sees the finished response:

MIDDLEWARE = [
    "django.middleware.gzip.GZipMiddleware",
    # ... your existing middleware ...
]

Django’s own documentation flags the caveat, and it’s worth reading before you switch this on globally: compressing a response that mixes attacker-controlled input with a secret can leak the secret via BREACH. In practice the pages large enough to cause this problem are public, cacheable and secret-free — which is precisely where gzip is safe.

Raise your tolerance too

Buffering fixes the cause. Switching worker class limits the damage of any future blocking write, and the two are worth doing together:

gunicorn MY_PROJECT.wsgi:application \
    --worker-class gthread \
    --workers 3 \
    --threads 4 \
    --bind unix:/var/www/MY_PROJECT/MY_PROJECT.sock

With the sync worker, a blocked write costs you a whole process. With gthread, it costs you one thread out of four and the process keeps serving. That is a bulkhead, not a repair — you still want the buffering — but it turns “the site is down” into “one request is stuck.”

Find the offending endpoints before your users do

You want to know which responses are large enough to be candidates. Drop this in and read the log for a week:

# MY_PROJECT/middleware.py
import logging
 
logger = logging.getLogger(__name__)
 
# Anything past this is big enough to block a worker in sendall().
LARGE_RESPONSE_BYTES = 1_000_000
 
 
class LargeResponseWarning:
    """Log responses large enough to be a write-timeout risk."""
 
    def __init__(self, get_response):
        self.get_response = get_response
 
    def __call__(self, request):
        response = self.get_response(request)
 
        # Streaming responses are fine - they don't buffer in one write.
        if not getattr(response, "streaming", False):
            size = len(response.content)
            if size > LARGE_RESPONSE_BYTES:
                logger.warning(
                    "Large response: %d bytes for %s", size, request.path
                )
 
        return response

Register it first — above GZipMiddleware if you added that above, so it measures the compressed bytes that actually go out. Django runs the response phase bottom-up, so the middleware at the top of the list is the last one to touch the response — which means it sees the actual bytes going out, after GZipMiddleware has had its turn:

MIDDLEWARE = [
    "MY_PROJECT.middleware.LargeResponseWarning",
    # ... your existing middleware ...
]

Almost always the answer is a list view with no pagination that was fine when the table had 400 rows. Paginating it is the actual root-cause fix, and it’s usually a two-line change.

Catching it while it happens

Everything so far is after the fact. While it is actually going on, one command shows you the whole thing. Run it on the box gunicorn is listening on:

ss -tn state established '( sport = :8000 )' | sort -k2 -n -r | head

Which gives you something like:

Recv-Q   Send-Q   Local Address:Port    Peer Address:Port
     0  1858240   10.10.0.7:8000        203.0.113.41:54388
     0  1642112   10.10.0.7:8000        203.0.113.88:54301
     0        0   10.10.0.7:8000        10.10.0.9:41122

Send-Q is data the kernel has accepted from your application but has not had acknowledged yet. Run the command twice, ten seconds apart. If those numbers sit there and don’t move, that connection is the one your worker is blocked on — no log correlation, no guessing which endpoint.

One caveat, which doubles as a restatement of the fix: this only shows you anything while gunicorn is bound to a TCP port. If it is on a unix socket behind a local nginx you will see nothing here — and you will not be getting ETIMEDOUT either, because a unix socket has no retransmission timer to expire.

Reading the two logs together

The last piece of confusion worth clearing up: this one event gets logged twice, from opposite ends, and the two entries look unrelated.

• Your proxy logs a 499 (nginx’s code for “client closed the request”) or a 502, and blames upstream.
• Your app logs TimeoutError: [Errno 110], and blames the network.

Neither is wrong, and neither is complete. Correlate them by timestamp and request path before you start tuning anything — if the 499s and the errno 110s line up on the same handful of large endpoints, you have confirmed the diagnosis and you know exactly which pages to shrink.

The short version

• [Errno 110] in sendall() is a write timeout. Your view already succeeded.
• A ~15 minute request duration is the TCP retransmission timer, not slow code.
• It surfaces on large responses only because small ones fit in the send buffer and fail invisibly.
• Raising --timeout hides the symptom and burns your worker pool.
• Buffer at the proxy, turn on gzip, paginate the huge pages, and use gthread so one stuck write can’t take a process with it.

There is a companion to this post: never run django.template.Template() on HTML you didn’t write — a different stack, the same trap. A traceback names the frame where things broke, which is rarely the frame that caused it.