Performance Management in Apache HTTPD (Worker MPM)

What an MPM actually is

Apache HTTPD does not have one concurrency model — it has a pluggable one. The Multi-Processing Module decides how the server accepts connections and how it maps them onto operating system processes and threads. Everything else in an Apache performance conversation depends on which MPM is loaded, which is why the first question is always httpd -V.

$ httpd -V | grep -i mpm
Server MPM:     worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)

# On Debian/Ubuntu the module is switched, not compiled:
$ a2dismod mpm_prefork && a2enmod mpm_worker && systemctl restart apache2

worker is the right default for a reverse proxy or static-content tier in front of an application server, which is the shape most Java estates take: HTTPD or IHS terminating TLS and proxying to Tomcat, JBoss or WebLogic over mod_proxy, mod_jk or mod_cluster. That is the deployment this article assumes.

How worker allocates threads

A worker-MPM Apache is a small hierarchy. One parent process, running as root, does nothing but bind the listening sockets and supervise children. It forks child processes, each of which creates a fixed number of worker threads plus one listener thread. The listener accepts a connection and hands it to an idle worker thread in the same process; that thread owns the connection until it closes.

httpd (parent, root)
 |
 +-- child 1 ── listener thread
 |               ├── worker thread 1 ─── connection ─── request ─── response
 |               ├── worker thread 2 ─── connection (keep-alive, idle)
 |               └── ... ThreadsPerChild threads
 |
 +-- child 2 ── listener thread
 |               └── ... ThreadsPerChild threads
 |
 +-- child N   (N varies between StartServers and ServerLimit)

Two consequences follow directly from that picture, and both are the source of most worker-MPM misconfiguration:

• The thread count per child is fixed at startup. Apache scales concurrency by forking or reaping whole processes, never by adding threads to an existing one. So capacity moves in units of ThreadsPerChild.
• A thread is occupied for the entire connection, not the request. Under worker, a client sitting on an idle keep-alive connection consumes a worker thread that could be serving someone else. This is precisely the problem the event MPM was built to solve.

The directives that matter

<IfModule mpm_worker_module>
    ServerLimit              16     # hard ceiling on child processes
    ThreadLimit              64     # hard ceiling on ThreadsPerChild
    ThreadsPerChild          64     # threads created in every child
    StartServers              4     # children forked at startup
    MinSpareThreads         128     # fork if idle threads fall below this
    MaxSpareThreads         384     # reap if idle threads exceed this
    MaxRequestWorkers      1024     # total concurrent connections served
    MaxConnectionsPerChild    0     # 0 = never recycle a child
</IfModule>

The relationship that governs everything is a single identity, and Apache will silently clamp your configuration if you break it:

The remaining directives are worth understanding individually:

• ThreadLimit and ServerLimit are ceilings evaluated once at startup and are not changeable on a graceful restart. Raising them requires a full stop and start, which is a genuine operational fact to plan around. Do not set them absurdly high "just in case" — the scoreboard is allocated from them and costs shared memory.
• ThreadsPerChild controls the granularity of scaling. Too low (say 8) and Apache forks a large number of processes, each with its own memory overhead and its own connection pool to the backend. Too high (say 250) and a single child crash takes a large slice of your capacity with it. 25 to 64 is the practical range for a proxy tier.
• MinSpareThreads / MaxSpareThreads drive the fork and reap decisions. Because a fork creates ThreadsPerChild threads at once, a MinSpareThreads smaller than ThreadsPerChild causes the server to oscillate — fork, overshoot the max spare, reap, dip below the min spare, fork again. Keep MinSpareThreads at roughly two children's worth and MaxSpareThreads comfortably above it.
• MaxConnectionsPerChild recycles a child after N connections. Zero is correct for a clean stack; a modest value (10000–50000) is a pragmatic mitigation if a third-party module leaks. It is a workaround, not a tuning parameter — recycling costs you a fork and cold backend connections.
• ListenBacklog sets the accept queue depth. It is capped by the kernel's net.core.somaxconn, so raising one without the other achieves nothing.

Sizing from measurement

MaxRequestWorkers is the single most consequential number in the file, and the two common ways of choosing it are both wrong. Copying a value from a blog ignores your workload; setting it very high "so we never queue" converts a manageable queue into a memory exhaustion event and an unresponsive server.

Derive it instead. Little's Law gives the concurrency a system needs:

concurrency = arrival rate × average residence time

# Example: 800 requests/sec, average 120 ms served end to end
concurrency = 800 × 0.120 = 96 concurrent requests in flight

Ninety-six is the steady-state requirement. Now apply headroom for burst and for backend slowdown — the case you actually care about, because that is when residence time balloons. A factor of two to three is normal, so roughly 200–300 workers. Then check it against memory, which is the hard limit:

# Resident memory per httpd child, in MB
$ ps -ylC httpd --sort:rss | awk '/httpd/ {sum+=$8; n++} END {print sum/n/1024" MB avg"}'

# Budget: (RAM available to httpd) / (memory per child) = affordable ServerLimit
# 8 GB for httpd, 90 MB per child  ->  ~88 children
# With ThreadsPerChild 64 that is far more than 300 workers, so memory is not
# the binding constraint here — the backend is.

For a proxy tier the binding constraint is almost never Apache itself. It is the application server behind it, and that leads to the most important sizing rule in this article:

Keep-alive: the hidden multiplier

Under worker, an idle keep-alive connection holds a worker thread. That makes KeepAliveTimeout a direct multiplier on your capacity requirement.

KeepAlive              On
MaxKeepAliveRequests   100
KeepAliveTimeout       3      # seconds — not the 15 in many default files

The arithmetic is unforgiving. At 500 new connections per second with a 15-second timeout, up to 7500 connections can be sitting idle but attached. No realistic MaxRequestWorkers covers that, so the server stops accepting new work while most of its threads are doing nothing.

Two to five seconds is the right range for a public-facing worker-MPM tier: long enough to reuse a connection across the assets of a single page view, short enough that abandoned connections release quickly. If your traffic genuinely benefits from long keep-alive — many small requests from a small number of clients — that is the argument for switching to the event MPM, where idle connections are handed back to a listener thread instead of pinning a worker.

Reading the scoreboard

mod_status with ExtendedStatus is the instrument panel. Without it you are tuning blind.

LoadModule status_module modules/mod_status.so
ExtendedStatus On

<Location "/server-status">
    SetHandler server-status
    Require ip 10.0.0.0/8            # never expose this publicly
</Location>

$ curl -s http://localhost/server-status?auto
BusyWorkers: 187
IdleWorkers: 13
Scoreboard: WWWWWKKKWW_WRWWWCWWWW.........

The scoreboard string is one character per worker slot, and the distribution tells you where time is going:

The decisive signal is BusyWorkers pinned at MaxRequestWorkers with the scoreboard full of W. That is not an Apache capacity problem — it is Apache waiting on something downstream, and raising MaxRequestWorkers will make it worse by pushing more concurrent work into an already saturated backend. Scrape this endpoint into Prometheus or your APM and alert on the ratio, not the absolute.

Aligning with the backend

In a proxied architecture, the numbers on both sides of the connector have to be chosen together. With mod_proxy, the pool is per child process, which is the detail that catches people out:

<Proxy "balancer://app">
    BalancerMember "http://10.0.2.11:8080" \
        min=4 max=64 smax=32 ttl=60 retry=30 \
        connectiontimeout=3 timeout=30
    BalancerMember "http://10.0.2.12:8080" \
        min=4 max=64 smax=32 ttl=60 retry=30 \
        connectiontimeout=3 timeout=30
</Proxy>

ProxyPass        "/app" "balancer://app/app" stickysession=JSESSIONID
ProxyPassReverse "/app" "balancer://app/app"

connectiontimeout and timeout deserve explicit values. Defaults inherited from Timeout are usually far too long, so a backend that has stopped responding holds Apache threads for the full duration instead of failing fast and letting the balancer retry elsewhere. A short connectiontimeout (2–3 s) with a timeout matched to your slowest legitimate request is the shape you want.

The same alignment applies to mod_jk, where connection_pool_size is likewise per child process and should be set to ThreadsPerChild, with connection_pool_timeout matched to Tomcat's connectionTimeout so neither side is holding half-dead sockets the other has already discarded.

Operating system limits

A well-tuned Apache still fails if the kernel and the process limits are not raised alongside it. Each connection is a file descriptor, and each proxied connection is two.

# File descriptors — the most common hard stop
$ cat /proc/$(pgrep -o httpd)/limits | grep 'open files'
# systemd unit override:
[Service]
LimitNOFILE=65535

# Accept queue: must be raised in the kernel AND in httpd.conf
$ sysctl -w net.core.somaxconn=4096
$ sysctl -w net.ipv4.tcp_max_syn_backlog=8192
ListenBacklog 4096

# Ephemeral ports for outbound proxy connections
$ sysctl -w net.ipv4.ip_local_port_range="10240 65535"
$ sysctl -w net.ipv4.tcp_fin_timeout=15

Two quick wins that are not thread tuning but usually matter more than it does: enable mod_deflate for text responses, and configure the TLS session cache. A stateless TLS handshake on every connection is measurable CPU, and it lands on exactly the tier you are trying to keep responsive.

SSLSessionCache        "shmcb:/var/run/httpd/sslcache(512000)"
SSLSessionCacheTimeout 300

AddOutputFilterByType DEFLATE text/html text/plain text/css \
                              application/javascript application/json

Symptoms and their causes

A tuning method that works

1. Establish the baseline. Measure requests per second and the latency distribution at the current configuration. Without a before, you cannot claim an after.
2. Find the actual bottleneck. Correlate BusyWorkers against backend response time. If workers rise while backend latency rises, Apache is a victim, not a cause.
3. Compute, do not copy. Derive MaxRequestWorkers from Little's Law, constrain it by the memory budget, and cap it at what the backend can absorb.
4. Change one variable per test. Load test with a realistic mix — JMeter or ab against your actual endpoints, not just / — and hold everything else constant.
5. Push to failure deliberately. You need to know where the knee is and how the server behaves past it. A configuration that degrades gracefully beats one with a marginally higher peak.
6. Instrument permanently. Ship mod_status metrics into your monitoring stack and alert on BusyWorkers / MaxRequestWorkers. Tuning is not a project you finish; traffic changes and the numbers age.

← Back to home