R1.1 — Connections and request handling: from the socket to your thread, and sizing the pools

Part of R1. In R1's story, the phone "sends an HTTPS call". Before a single header of that call moves, the phone and the server have to agree to talk: they open a connection. And once the bytes arrive, something has to get them from the network card into your controller method. This part covers both. Much of what follows in the pack depends on it: pools exist because connections are expensive (R11), the first wall in scaling is a pool (R13), timeouts guard each stage of the trip (R14), and a chat server's state is mostly its connections (R18).

What a connection is

A TCP connection is an agreement between two programs, on two machines, to exchange a reliable, ordered stream of bytes. Nothing physical is created. Each machine's operating system kernel keeps a small record about the conversation: which bytes have been sent, which have been acknowledged, how much the other side can accept, and buffers for data waiting to be read or sent. The connection is those two records, one at each end, kept in step by the packets that flow between them.

That record-keeping is what TCP buys you. Lost packets are resent, packets that arrive out of order are put back in order, and a fast sender is slowed down for a slow receiver. Your code just reads and writes a stream.

Watch a connection being opened, then used. Each step shows one message and what each machine now knows.

UDP is the alternative without a connection: each packet travels alone, with no ordering and no retry. DNS lookups use it, Zoom's media uses it because late audio is worthless (R18), and HTTP/3 builds its own reliability on top of it.

Ports and sockets

A port picks one program on a machine. An IP address gets a packet to the right machine. A port, a number from 0 to 65535, gets it to the right program on that machine. Servers listen on well-known ports — 443 for HTTPS, 5432 for Postgres, 6379 for Redis, 9092 for Kafka. The client side uses an ephemeral port, a temporary number its operating system picks for that one connection.

A socket is the handle a program uses to send and receive. On Linux it is a file descriptor, the same kind of handle as an open file. Writing to it hands bytes to the kernel; reading from it takes bytes the kernel has received.

Four numbers identify a connection: the client's address and port, and the server's address and port.

The server has one listening socket on port 443, and a separate connected socket for every connection. The listening socket only accepts new connections. Each accepted connection gets its own socket, and all of them share local port 443; the kernel tells them apart by the client's address and port. That is how one server process holds tens of thousands of connections on a single port.

Watch the sockets appear on the server as two phones connect.

Then follow one incoming packet, and see how the kernel finds the one socket it belongs to.

The trap: "a server can only hold 65,535 connections, because that is how many ports there are."

A server's connections all use the same port, so ports do not limit it. File descriptors, memory and threads do. The port limit bites the client side. One machine connecting to one destination address and port has only its ephemeral range to choose from, about 28,000 ports on a default Linux. A closed connection keeps its port reserved for about a minute afterwards, in a state called TIME_WAIT. A service that opens a fresh connection for every call to a downstream service can run out of ports under load. The fix is to reuse connections.

Watch it happen on the client, then see how a pool fixes it.

What a connection costs

Reuse: open once, use many times

Every row in that table is paid again for every new connection, so the rule is simple: reuse connections wherever possible.

What people mean by an "HTTP connection". HTTP has no connection of its own. It is a format for requests and responses — a method, a path, headers, a body — and it needs something underneath to carry those bytes reliably and in order. An "HTTP connection" is therefore a TCP connection, usually wrapped in TLS, that is being used to carry HTTP messages. Opening one means a TCP handshake and then a TLS handshake. The requests come and go over it afterwards. HTTP/3 is the exception: there, the connection is a QUIC connection over UDP, with TLS built in.

Reuse starts with the client. The side that opens a connection — the client — chooses whether to keep it for the next request or open a new one each time. (The server can still close a connection that sits idle too long; the keep-alive timeout later in this part does exactly that.) "Client" here is a role, not a kind of machine: the phone is the client when it calls your API, and your payment service becomes a client the moment it calls the fraud-check service. In Java, the code that plays that role is an HTTP client library: Apache HttpClient, Spring's RestClient or WebClient, or the JDK's HttpClient.

The simplest reuse is keep-alive: one connection, kept open for the next request. HTTP/1.1 does this by default. Compare the first request on a new connection with the next one on the same connection.

One kept-alive connection is not enough for a busy service. HTTP/1.1 carries one request at a time on a connection. A payment service with 200 requests in flight, each calling the fraud-check service, needs several of those calls in progress at once — so it needs several open connections to the same destination, and something to hand them out.

HTTP client connection pools

That something is the HTTP client connection pool, built into the HTTP client library. It keeps connections to each destination open and lends each one to a single call at a time. A call that finds an idle connection skips the TCP and TLS handshakes, and a connection that is returned rather than closed never sits in TIME_WAIT.

What the pool holds. Open connections, grouped by destination — scheme, host and port, such as https://fraud.internal:443. Apache HttpClient calls each destination a route. A connection is either leased (one call is using it) or idle (open, waiting for the next call to the same destination). Two limits cap the pool: one per route and one in total. Apache HttpClient 5 defaults to 5 per route and 25 in total.

flowchart TD
  A[Request thread calls fraud.internal] --> B{Idle connection<br/>to that route?}
  B -- yes --> C[Lease it<br/>validate if idle a while]
  B -- no --> D{Below per-route<br/>and total limit?}
  D -- yes --> E[Open a new one<br/>TCP + TLS handshakes]
  D -- no --> F[Wait for one to be returned<br/>up to connection-request timeout]
  F -- returned in time --> C
  F -- timed out --> G[Fail the call]
  C --> H[Send request, read response]
  E --> H
  H --> I[Return it to the pool<br/>kept alive for the next call]

1. Lease. The calling thread asks the pool for a connection to the route. If an idle one exists, it gets it at once — no handshakes.
2. Open, if allowed. If none is idle and the route and total are both below their limits, the pool opens a new connection, paying the handshakes once.
3. Wait, if not. If the route is at its limit, the thread waits for another request to return a connection, for up to the connection-request timeout. This is the queue that hides behind a small per-route limit: nothing fails, calls just get slower.
4. Use. The request is sent and the response read. The connection is leased for exactly that long.
5. Return. The connection goes back to the pool, still open, ready for the next call. It is closed only if the server asked to close it, if it broke, or if it is evicted.

The response body must be fully read or closed, or the connection never comes back.

A leased connection returns to the pool only when the response is consumed. Code that reads the status code and ignores an error body can leak a connection on every failure. After a few leaks, all five connections for the route are stuck and every call waits until it times out.

The limits and timeouts that matter in production are under "Tuning the HTTP client pool" further down.

The same idea appears wherever connections are expensive. HTTP keep-alive sends many requests over one connection. Database connection pools keep a set of database connections open and lend them out in exactly the same way (R11). HTTP/2 goes further and carries many requests over one connection at the same time, as the next table shows.

How HTTP versions share a connection

Once a connection is open, the HTTP version decides how many requests can use it at the same time.

Keep three counts apart, because they differ by orders of magnitude. A connection lives from handshake to close and can carry many requests. A request is one HTTP exchange. A worker thread is borrowed from a pool for one request and returned.

From the network card to your controller: one request through Tomcat

Spring Boot's default web server is an embedded Tomcat with the NIO connector. The connector is the part of Tomcat that talks to the operating system's sockets: it owns the acceptor, the poller and the pool of worker threads. A worker thread then calls into Tomcat's servlet container, where Spring and your controller run. So the layers, from the bottom up, are the kernel, the connector, and your code. Follow the payment request from the moment its packets reach the server. Each stage hands the work to the next, and this order is the real order of events:

1. The kernel completes the handshake on its own. A half-open connection waits in the SYN queue. Once the client's final ACK arrives, the connection moves to the accept queue (also called the backlog), ready to be picked up. No Java code has run yet.
2. Tomcat's acceptor thread takes it. It calls accept(), which removes the connection from the accept queue and returns a new socket. The acceptor switches the socket to non-blocking mode and hands it to the poller.
3. The poller watches every open connection with one thread. It uses Java NIO's selector — epoll on Linux — to ask the kernel which of thousands of sockets has bytes waiting. An idle connection costs this thread nothing.
4. A worker thread processes the request. When bytes arrive, the poller hands the socket to a thread from Tomcat's pool (you will see names like http-nio-8080-exec-7 in thread dumps). That thread reads the bytes, parses the request line and headers, builds the HttpServletRequest, and runs the servlet filter chain (Spring Security lives here), then DispatcherServlet, which calls your controller.
5. Your code runs entirely on that thread. There is no separate application thread unless you create one. The controller is a single object that many worker threads run at the same moment, which is why it must not keep per-request data in its fields. When the service calls the database through JDBC, the thread waits — blocked — until the answer comes back.
6. The response goes back the same way. The thread writes the response into the socket's send buffer and returns to the pool. The kernel transmits the bytes. The socket goes back to the poller to wait for the next request on the same connection.

The animation below starts from a connection that is already open — the handshake and the accept queue (steps 1 and 2) only matter while a connection is being set up. It follows one request from the socket to your controller and back. It shows five open connections; a real poller watches thousands.

Which threads wait, and on what. Each thread in the journey has one job, and only one of them runs your code:

The settings that bound each stage, with Spring Boot's property names:

The design point of NIO is that thousands of connections share a few hundred threads. A thread is needed only while a request is actually being processed. An open connection with nothing to say costs a socket and some buffers, not a thread. The older blocking connector gave every connection its own thread for as long as it stayed open.

Tomcat's threads, and what happens on one connection

A running Spring Boot service has more threads than most people expect. In a thread dump of the web tier you will find three kinds:

A request on a connection is handled like this. HTTP/1.1 carries one request at a time on a connection. While a request is being processed, its socket is with a worker thread, not the poller. When the response is written, the socket goes back to the poller, and the next request on that connection may be handled by a completely different worker thread. Threads belong to requests; connections only borrow them.

There is one way to give the worker thread back before the response is ready. If a controller returns a CompletableFuture or a DeferredResult, Spring MVC releases the Tomcat thread and finishes the response later, on whichever thread completes the future. That frees Tomcat's pool, but the waiting has to happen somewhere else — usually in another pool.

Watch the hand-off: the worker thread is released early, an application thread does the blocking work, and a worker is needed again only to write the response.

Every pool a request touches

A thread pool holds threads; a connection pool holds open connections. A request in a typical payment service touches several of each, and any one of them can be the limit. Several ship with defaults that are much smaller than people assume.

The traps hide in the small defaults.

A service with 200 Tomcat threads that calls one downstream API through Apache HttpClient can run only 5 of those calls at a time, because of the per-destination limit; the other 195 threads queue for an HTTP connection. All scheduled jobs share one thread, so one slow job delays every other. The @Async pool never grows past 8, because its queue never fills. And blocking I/O inside supplyAsync without an executor ties up the common pool that parallel streams elsewhere in the JVM also use.

Tuning the HTTP client pool

The pool introduced under "What a connection costs" works without any configuration, but its defaults are generic: a small pool and very long waits. A payment service that calls other services needs to make five decisions. Each subsection below starts with the idea in plain words, then gives the settings and values to use as a reference. The checklist at the end puts all of them in one table.

How big: the two limits

In plain words. The pool can only lend out as many connections as its limits allow. If 30 requests want to call the fraud API at the same moment and the limit for that destination is 5, then 5 calls run and 25 wait in line. Nothing fails; everything just gets slower.

For reference. Size the per-route limit from Little's Law: calls per second to that destination × how long each call takes, times about 2 for headroom. Set the total to at least the sum of the per-route limits you expect to use at once. The worked example in the next section does this for the fraud API: 200 calls a second × 40 ms = 8 in use, so 16 per route. With the default of 5, the instance can never do more than 5 ÷ 0.040 s = 125 calls a second.

A limit that is too small never shows up as an error. It shows up as latency, and as threads waiting for connections. Look for it in the "pending" metric under "How to watch it" below.

How long to wait: three timeouts

In plain words. An outgoing call can get stuck in three different places, and each has its own timeout:

1. Waiting for the pool to hand over a connection, because they are all in use.
2. Opening a new connection, if the pool has to make one: the TCP handshake.
3. Waiting for the response, after the request has been sent.

Step through where each one applies:

For reference. Set all three. Every second a call waits, it holds a Tomcat worker thread, so a call allowed to wait three minutes can take your whole service down with it. Make the outgoing timeouts shorter than the timeout of the request that triggered them, so your service answers its own caller before that caller gives up. R14 covers timeouts, retries and circuit breakers in depth.

How to stay healthy: connections that die while idle

In plain words. A pooled connection can sit unused between calls. While it sits there, something in the path may quietly close it: the server, a load balancer or a NAT gateway. An AWS Application Load Balancer, for example, closes connections that have been idle for 60 seconds. The pool doesn't know, so it still lists the connection as available. The next call that borrows it fails with "connection reset" or "no response".

The fix is to retire connections before anything else does, and to check old ones before use:

Which client: know what is actually pooling

In plain words. "The HTTP client" in a Spring service can be one of several libraries, and they pool differently. The most common surprise is a RestTemplate created with new RestTemplate(): it has no real pool and no timeouts at all.

HTTP/2 changes the arithmetic. One HTTP/2 connection carries many requests at once, so a client may need only one or two connections per destination. The limit that matters becomes how many streams the server allows on each connection, often around 100.

How to watch it: the metrics that matter

In plain words. A pool has three numbers worth watching: how many connections are in use, how many are idle, and how many callers are waiting for one. Waiting callers mean the pool is too small, or the downstream is too slow.

Micrometer publishes these for Apache HttpClient 5 when the connection manager is bound to its registry:

Pair them with the downstream's latency. If pending rises while the downstream is fast, raise the limit. If pending rises because the downstream got slow, a bigger pool only sends more traffic to a struggling service — that is a job for timeouts and a circuit breaker (R14).

Reference: the whole configuration in one place

The fraud-service client from the worked example, with every decision above made explicitly:

PoolingHttpClientConnectionManager pool = PoolingHttpClientConnectionManagerBuilder.create()
    .setMaxConnPerRoute(16)                                  // how big: concurrent calls to fraud.internal
    .setMaxConnTotal(40)                                     // how big: across every downstream
    .setDefaultConnectionConfig(ConnectionConfig.custom()
        .setConnectTimeout(Timeout.ofSeconds(1))             // how long: TCP handshake
        .setValidateAfterInactivity(TimeValue.ofSeconds(5))  // healthy: check long-idle connections
        .setTimeToLive(TimeValue.ofMinutes(5))               // healthy: retire old connections
        .build())
    .build();

CloseableHttpClient http = HttpClients.custom()
    .setConnectionManager(pool)
    .setDefaultRequestConfig(RequestConfig.custom()
        .setConnectionRequestTimeout(Timeout.ofMilliseconds(200))  // how long: waiting for the pool
        .setResponseTimeout(Timeout.ofSeconds(2))                  // how long: waiting for the answer
        .build())
    .evictIdleConnections(TimeValue.ofSeconds(30))           // healthy: below the ALB's 60 s idle timeout
    .build();

RestClient fraud = RestClient.builder()
    .requestFactory(new HttpComponentsClientHttpRequestFactory(http))  // which client: explicit
    .baseUrl("https://fraud.internal")
    .build();

new PoolingHttpClientConnectionManagerMetricsBinder(pool, "fraud").bindTo(meterRegistry);  // how to watch

Rules of thumb for HTTP client pools

• Build the client once and share it; never create a new client or pool per request.
• Size each route from calls per second × call duration, then double it.
• Set all three timeouts, each shorter than the caller's own deadline.
• Retire idle connections before the load balancer or server does.
• Always read or close the response body, so the connection goes back to the pool.
• Alert on pending acquisitions, not just on errors.

Sizing: work backwards from the scarcest resource

One law sizes every pool: Little's Law. The average number of things in use equals the rate they arrive times how long each is held.

in use  =  arrival rate  ×  time each one is held

Applied to each pool, "time held" means something different: for Tomcat threads, the whole request; for Hikari, the time a request holds a connection; for the HTTP client, the duration of the downstream call. Then add headroom for bursts and slow moments, typically 1.5 to 2 times.

For a thread pool, the kind of work matters. A thread doing pure computation keeps a core busy, so more threads than cores just take turns. A thread waiting on I/O uses no CPU, so you need more of them to keep the cores busy. Brian Goetz's rule from Java Concurrency in Practice says how many:

threads to keep the CPUs busy  =  cores  ×  (1 + wait time ÷ compute time)

For CPU-bound work that is about the number of cores. For a typical web request that mostly waits, it is many times that.

Size in this order, from the scarcest resource outwards: the database, then the database pool, then the outgoing HTTP pools, then the Tomcat threads, then a check on connections. A pool larger than what sits behind it only moves the queue.

A worked example: the payment-initiation service

The service runs as 6 instances with 2 vCPUs each, behind a load balancer, on an Aurora writer with 8 vCPUs. At peak it takes 1,200 requests a second, so 200 per instance. Each request spends about:

1. The database. A relational database runs efficiently only a limited number of queries at once; past that, more connections contend for the same cores and locks and everything slows. A common starting point, from the HikariCP and PostgreSQL guidance, is about twice the database's cores — here around 16. Across the fleet, connections in use on average = 1,200 × 0.015 s = 18. That is at the edge, worth stating aloud, and a reason to keep transactions short.
2. The Hikari pool per instance. In use = 200 × 0.015 s = 3. With 2× headroom, set 6. The fleet then holds 6 × 6 = 36 connections, most of them idle at any moment, which Aurora handles easily. If autoscaling can take the fleet to 20 instances, that becomes 120 connections — and more with a bigger pool — which is R13's first wall. A proxy (RDS Proxy, PgBouncer) fixes that, not a bigger pool (R11).
3. The HTTP client pool to the fraud service. In use = 200 × 0.040 s = 8. With headroom, allow 16 per destination. With the default of 5, the most this instance could ever do is 5 ÷ 0.040 s = 125 requests a second, below the 200 it needs. Nothing would show up as an error. Latency would simply climb.
4. Tomcat threads. Requests in flight = 200 × 0.060 s = 12 on average. Goetz's rule gives 2 × (1 + 55 ÷ 4) ≈ 30 to keep both cores busy, and CPU use is 200 × 4 ms = 0.8 cores, 40% of the instance. A value around 50 covers bursts and slow tails. The default of 200 also works at normal load. The choice only matters under overload: a lower limit sheds work sooner, while a higher one lets more threads pile up waiting on a 6-connection pool.
5. Connections. Behind a load balancer, the balancer holds the clients' connections and keeps a modest set of reused connections to each instance — roughly the requests in flight plus some idle keep-alive. That is tens or hundreds, far below Tomcat's 8,192. Leave max-connections and accept-count at their defaults.

How to set the maximum number of connections

Unlike the pools, the connection limit is not derived from Little's Law. It is a ceiling on memory and file descriptors, and it matters only when clients hold connections open without sending requests: WebSocket and SSE servers (R18), long polling, or IoT devices connecting directly. Then size it from what each connection costs:

Raising the connection limit never adds capacity. It only lets more clients wait. On an ordinary request–response API, a larger max-connections or accept-count hides overload for longer and turns fast failures into slow timeouts.

Try the numbers yourself. The calculator opens on the worked example with the default HTTP limit, so two checks fail: the database is at its edge, and a limit of 5 caps each instance at 125 requests a second. Press Fixed to shorten the transaction and raise the limit, then Scale out and watch the fleet's database connections grow. Change any time and see which pool becomes the limit first.

Sizing rules to say out loud

• Measure the time each request holds each resource; Little's Law turns those times into pool sizes.
• Start from the scarcest resource, usually the database, and size every pool in front of it no larger than it can use.
• Multiply every per-instance pool by the maximum instance count, and check the total against the database.
• Check the small defaults: HTTP client per-destination limits, the single scheduler thread, the 8-thread @Async pool, the common pool.
• Re-measure after each change. A pool that is too small shows up as pending acquisitions; a pool that is too big shows up as slower queries.

When it overloads: the failure travels outwards

Run the stages in reverse and you get the order in which an overloaded service fails. Suppose the database slows from 15 ms to 2 seconds per query:

1. Worker threads fill up. The connection pool has 10 connections, so only 10 queries run at a time, and the service finishes at most 10 ÷ 2 s = 5 requests a second, whatever the CPU says (Little's Law, R13). Every other request takes a worker thread and waits for a database connection. At 500 requests a second, all 200 threads are taken in under a second.
2. Requests queue inside Tomcat. New requests are read and wait for a free thread. Latency climbs; clients' read timeouts start firing (R14).
3. Connections pile up. Waiting clients hold their connections open, and new ones keep arriving, until Tomcat reaches maxConnections. Then the acceptor stops accepting.
4. The kernel's accept queue fills. Completed handshakes wait there, up to acceptCount.
5. The kernel starts dropping new connections. With the accept queue full, Linux by default silently drops the handshake. The client resends after a second, then longer, and finally reports a connect timeout.

Watch the overload spread outwards, one stage per step.

A slow dependency shows up first as busy threads and last as connect timeouts at the TCP layer, two hops away from the cause. Read the stack from the bottom up.

The defences belong in R14: short timeouts on the database call so threads are freed, a bulkhead so one slow dependency cannot take every thread, and load shedding with a fast 503 at the edge before queues fill silently.

Virtual threads: removing the thread ceiling, not the bottleneck

Java 21 added virtual threads: threads managed by the JVM rather than the operating system, cheap enough to create one per request. When a virtual thread blocks on I/O, such as a JDBC call, the JVM parks it and frees the small set of real threads underneath (the carrier threads, about one per CPU core) to run other work. In Spring Boot 3.2 and later, setting spring.threads.virtual.enabled=true makes Tomcat run every request on its own virtual thread.

See where the waiting goes in each case.

Virtual threads help when waiting is the problem, not when the dependency is saturated. A service that calls three slow external APIs gets far more concurrency from the same machine. A service whose database can run only 10 queries at once still runs 10 queries at once. The other requests now queue on the connection pool instead of on threads, so the pool's connection-timeout (R11) and an explicit concurrency limit — a semaphore or a bulkhead (R14) — become the brakes.

One caveat for interviews: on Java 21 to 23, a virtual thread that blocks inside a synchronized block pins its carrier, so the carrier cannot run anything else. Java 24 removed that limitation. A fully non-blocking stack — Spring WebFlux on Netty — is the other route to the same goal, and R17's gateway design discusses when it is worth the rewrite.

Rules of thumb for connections and request handling

• Reuse connections everywhere: keep-alive towards clients, pools towards databases and downstream services.
• Size the worker pool and the database pool from Little's Law, not from the defaults.
• Set a timeout at every stage, with each inner timeout shorter than the one outside it.
• Watch busy threads, pending pool acquisitions and listen-queue overflows together; any one alone misleads.
• Refuse early and visibly — rate limits and fast 503s at the edge — rather than letting queues fill until the kernel drops connections.

What to say in the interview

"Connections are expensive to open, so I reuse them — keep-alive towards clients, pools towards the database and downstream services. I size every pool from Little's Law, starting at the database and working outwards, and check the fleet's total connections against what the database can run well. Under overload a slow query exhausts Tomcat's threads long before it exhausts CPU, so each dependency gets a timeout and a bulkhead, and the edge sheds load early rather than letting the kernel drop connections."