Apache eats all my memory!

I have an Apache 2.0.54 server on a Linux (2.6.11) box which has 1Gig
RAM and an additional 1Gig swap space. The server handles a lot of
people downloading sermons from our church website (which are generally
about 16-18Meg MP3 files), but I can't figure out how to keep the server
from running out of memory.

Here's my Apache2 prefork configuration:
------------------------------------------------------------------------
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 20
MaxRequestsPerChild 5000
</IfModule>


And here's what the Apache "/server-status" URL showed earlier today (I
had just restarted the server, but it immediately filled up with
download requests, all from the same guy, apparently using a download
accelerator judging by the duplicate requests):
------------------------------------------------------------------------
Srv PID M CPU Req Request
0-0 15822 W 0.48 0 GET /out/181.mp3 HTTP/1.1
1-0 15823 W 0.00 1742573500 GET /out/388.mp3 HTTP/1.1
2-0 15824 W 0.00 1742573499 GET /out/238.mp3 HTTP/1.1
3-0 15825 W 0.00 1742573499 GET /out/504.mp3 HTTP/1.1
4-0 15826 W 0.00 1742573496 GET /out/388.mp3 HTTP/1.1
5-0 15832 W 0.00 1742572495 GET /out/801.mp3 HTTP/1.1
6-0 15834 W 0.00 1742571493 GET /out/504.mp3 HTTP/1.1
7-0 15835 W 0.00 1742571489 GET /out/504.mp3 HTTP/1.1
8-0 15838 W 0.00 1742570476 GET /out/388.mp3 HTTP/1.1
9-0 15839 W 0.00 1742570484 GET /out/504.mp3 HTTP/1.1
10-0 15840 W 0.60 0 GET /out/238.mp3 HTTP/1.1
11-0 15841 W 0.00 1742570477 GET /out/388.mp3 HTTP/1.1
12-0 15846 W 0.25 0 GET /out/181.mp3 HTTP/1.1
13-0 15847 W 0.00 1742569347 GET /out/181.mp3 HTTP/1.1
14-0 15848 W 0.00 1742568761 GET /out/801.mp3 HTTP/1.1
15-0 15849 W 0.00 1742568761 GET /out/801.mp3 HTTP/1.1
16-0 15852 W 0.19 0 GET /out/181.mp3 HTTP/1.1
17-0 15853 W 0.17 0 GET /out/801.mp3 HTTP/1.1
18-0 15854 W 0.22 0 GET /out/504.mp3 HTTP/1.1
19-0 15855 W 0.28 0 GET /server-status HTTP/1.1


And here's a portion of what "top" showed at the same time:
------------------------------------------------------------------------
top - 18:09:59 up 64 days, 7:08, 3 users, load avg: 21.62, 10.57, 4.70

Tasks: 154 total, 1 running, 143 sleeping, 1 stopped, 9 zombie
Cpu(s): 0.8% us, 2.3% sy, 0.0% ni, 0.0% id, 96.3% wa, 0.3% hi, 0.2% si
Mem: 1034276k total, 1021772k used, 12504k free, 6004k buffers
Swap: 1030316k total, 985832k used, 44484k free, 83812k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

15846 apache 16 0 132m 89m 1968 S 0.3 8.9 0:01.46 apache2

15840 apache 17 0 130m 83m 2008 D 0.0 8.3 0:00.90 apache2

15849 apache 16 0 120m 82m 1968 S 0.3 8.1 0:01.02 apache2

15852 apache 16 0 120m 81m 1968 S 0.3 8.1 0:00.91 apache2

15848 apache 16 0 109m 73m 2008 S 0.3 7.2 0:00.85 apache2

15855 apache 16 0 107m 70m 2076 D 0.3 7.0 0:00.76 apache2

15822 apache 17 0 179m 55m 1968 D 0.3 5.5 0:00.88 apache2

15854 apache 16 0 98024 55m 1968 D 0.0 5.5 0:00.58 apache2

15853 apache 18 0 98.9m 53m 2000 S 0.0 5.3 0:00.51 apache2

15847 apache 17 0 86884 52m 1968 D 0.0 5.2 0:00.42 apache2

15841 apache 17 0 110m 36m 1964 D 0.3 3.6 0:00.64 apache2

15826 apache 17 0 173m 20m 1968 D 0.0 2.0 0:00.57 apache2

15825 apache 16 0 97.7m 19m 1968 D 0.0 1.9 0:00.36 apache2

15834 apache 16 0 117m 14m 1968 D 0.3 1.5 0:00.42 apache2

15839 apache 17 0 115m 12m 1968 D 0.0 1.2 0:00.40 apache2

15838 apache 15 0 182m 12m 1968 D 0.0 1.2 0:00.59 apache2

15823 apache 16 0 180m 11m 1968 D 0.0 1.1 0:00.65 apache2

15824 apache 15 0 103m 9980 1968 D 0.0 1.0 0:00.27 apache2

15832 apache 16 0 116m 9112 1968 D 0.0 0.9 0:00.29 apache2

15835 apache 16 0 162m 8844 1968 D 0.0 0.9 0:00.41 apache2
(everything else listed on "top" below this was less than 0.5 for %MEM)


The memory usage swelled very fast as the download requests came in, and
based on previous experience, the server would have slowed to a crawl
and possible crashed as it tried to save itself if I hadn't run "killall
apache2" at this point.

So it seems like this guy's 19 download requests are enough to pretty
much exhaust my 1 Gig of physical RAM and 1 Gig of swap space. That
just doesn't seem right. What am I doing wrong that so few download
requests can bring the server to its knees? How can I fix this
configuration?

Thanks to anyone who can help!

>I have an Apache 2.0.54 server on a Linux (2.6.11) box which has 1Gig
> RAM and an additional 1Gig swap space. The server handles a lot of people
> downloading sermons from our church website (which are generally about
> 16-18Meg MP3 files), but I can't figure out how to keep the server from
> running out of memory.
>
> Here's my Apache2 prefork configuration:
> ------------------------------------------------------------------------
> <IfModule prefork.c>
> StartServers 5
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients 20
> MaxRequestsPerChild 5000
> </IfModule>
>
>
> And here's what the Apache "/server-status" URL showed earlier today (I
> had just restarted the server, but it immediately filled up with download
> requests, all from the same guy, apparently using a download accelerator
> judging by the duplicate requests):
> ------------------------------------------------------------------------
> Srv PID M CPU Req Request
> 0-0 15822 W 0.48 0 GET /out/181.mp3 HTTP/1.1
> 1-0 15823 W 0.00 1742573500 GET /out/388.mp3 HTTP/1.1
> 2-0 15824 W 0.00 1742573499 GET /out/238.mp3 HTTP/1.1
> 3-0 15825 W 0.00 1742573499 GET /out/504.mp3 HTTP/1.1
> 4-0 15826 W 0.00 1742573496 GET /out/388.mp3 HTTP/1.1
> 5-0 15832 W 0.00 1742572495 GET /out/801.mp3 HTTP/1.1
> 6-0 15834 W 0.00 1742571493 GET /out/504.mp3 HTTP/1.1
> 7-0 15835 W 0.00 1742571489 GET /out/504.mp3 HTTP/1.1
> 8-0 15838 W 0.00 1742570476 GET /out/388.mp3 HTTP/1.1
> 9-0 15839 W 0.00 1742570484 GET /out/504.mp3 HTTP/1.1
> 10-0 15840 W 0.60 0 GET /out/238.mp3 HTTP/1.1
> 11-0 15841 W 0.00 1742570477 GET /out/388.mp3 HTTP/1.1
> 12-0 15846 W 0.25 0 GET /out/181.mp3 HTTP/1.1
> 13-0 15847 W 0.00 1742569347 GET /out/181.mp3 HTTP/1.1
> 14-0 15848 W 0.00 1742568761 GET /out/801.mp3 HTTP/1.1
> 15-0 15849 W 0.00 1742568761 GET /out/801.mp3 HTTP/1.1
> 16-0 15852 W 0.19 0 GET /out/181.mp3 HTTP/1.1
> 17-0 15853 W 0.17 0 GET /out/801.mp3 HTTP/1.1
> 18-0 15854 W 0.22 0 GET /out/504.mp3 HTTP/1.1
> 19-0 15855 W 0.28 0 GET /server-status HTTP/1.1
>
>
> And here's a portion of what "top" showed at the same time:
> ------------------------------------------------------------------------
> top - 18:09:59 up 64 days, 7:08, 3 users, load avg: 21.62, 10.57, 4.70
> Tasks: 154 total, 1 running, 143 sleeping, 1 stopped, 9 zombie
> Cpu(s): 0.8% us, 2.3% sy, 0.0% ni, 0.0% id, 96.3% wa, 0.3% hi, 0.2% si
> Mem: 1034276k total, 1021772k used, 12504k free, 6004k buffers
> Swap: 1030316k total, 985832k used, 44484k free, 83812k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 15846 apache 16 0 132m 89m 1968 S 0.3 8.9 0:01.46 apache2
> 15840 apache 17 0 130m 83m 2008 D 0.0 8.3 0:00.90 apache2
> 15849 apache 16 0 120m 82m 1968 S 0.3 8.1 0:01.02 apache2
> 15852 apache 16 0 120m 81m 1968 S 0.3 8.1 0:00.91 apache2
> 15848 apache 16 0 109m 73m 2008 S 0.3 7.2 0:00.85 apache2
> 15855 apache 16 0 107m 70m 2076 D 0.3 7.0 0:00.76 apache2
> 15822 apache 17 0 179m 55m 1968 D 0.3 5.5 0:00.88 apache2
> 15854 apache 16 0 98024 55m 1968 D 0.0 5.5 0:00.58 apache2
> 15853 apache 18 0 98.9m 53m 2000 S 0.0 5.3 0:00.51 apache2
> 15847 apache 17 0 86884 52m 1968 D 0.0 5.2 0:00.42 apache2
> 15841 apache 17 0 110m 36m 1964 D 0.3 3.6 0:00.64 apache2
> 15826 apache 17 0 173m 20m 1968 D 0.0 2.0 0:00.57 apache2
> 15825 apache 16 0 97.7m 19m 1968 D 0.0 1.9 0:00.36 apache2
> 15834 apache 16 0 117m 14m 1968 D 0.3 1.5 0:00.42 apache2
> 15839 apache 17 0 115m 12m 1968 D 0.0 1.2 0:00.40 apache2
> 15838 apache 15 0 182m 12m 1968 D 0.0 1.2 0:00.59 apache2
> 15823 apache 16 0 180m 11m 1968 D 0.0 1.1 0:00.65 apache2
> 15824 apache 15 0 103m 9980 1968 D 0.0 1.0 0:00.27 apache2
> 15832 apache 16 0 116m 9112 1968 D 0.0 0.9 0:00.29 apache2
> 15835 apache 16 0 162m 8844 1968 D 0.0 0.9 0:00.41 apache2
> (everything else listed on "top" below this was less than 0.5 for %MEM)
>
>
> The memory usage swelled very fast as the download requests came in, and
> based on previous experience, the server would have slowed to a crawl and
> possible crashed as it tried to save itself if I hadn't run "killall
> apache2" at this point.
>
> So it seems like this guy's 19 download requests are enough to pretty much
> exhaust my 1 Gig of physical RAM and 1 Gig of swap space. That just
> doesn't seem right. What am I doing wrong that so few download requests
> can bring the server to its knees? How can I fix this configuration?
>
> Thanks to anyone who can help!

What distribution of Linux are you running? 2.6.11 sounds very new.

> What distribution of Linux are you running? 2.6.11 sounds very new.

Gentoo, although I've had this problem both with 2.4 and 2.6 series kernels.

George Adams wrote:
> I have an Apache 2.0.54 server on a Linux (2.6.11) box which has 1Gig
> RAM and an additional 1Gig swap space. The server handles a lot of
> people downloading sermons from our church website (which are generally
> about 16-18Meg MP3 files),

If you are servicing byterange requests, apache will load the
entire response into memory (it's a bug that's due for fixing,
round tuits and consensus permitting). So for files of that kind
of size, it's trouble.

RFC2616 permits a server to just disable byteranges, and clients
don't have a problem with it. So that's your solution.

--
Nick Kew

> If you are servicing byterange requests, apache will load the
> entire response into memory (it's a bug that's due for fixing,
> round tuits and consensus permitting). So for files of that kind
> of size, it's trouble.
>
> RFC2616 permits a server to just disable byteranges, and clients
> don't have a problem with it. So that's your solution.

Nick, is there a workaround for this in Apache 2.0.54, or will I just
have to wait for the 2.1 branch to become stable?

Plus, it seems like even if all 20 of my Apache processes are reading an
18Meg file, that's still only 20x18=360Meg - quite a lot, but nowhere
near the 2Gig limit of my system.

Does the bug you're describing result in Apache using up more memory
than the actual size of the file it's trying to process?

Thanks for the help.

Search on Wiki for "tiny web servers"?

Will using a CacheFile and zlib (php & zlib.output_compression) not prevent
this or speedup the download? Or can you not put the file on a sub domain
that has a limit to say five downloads at a time?

Regards
Nik
http://www.ads4links.com


"D. Stussy" <kd6lvw@bde-arc.ampr.org> wrote in message
news:Pine.LNX.4.62.0508150038350.87@kd6lvw.ampr.or g...
> On Mon, 8 Aug 2005, George Adams wrote:
> > I have an Apache 2.0.54 server on a Linux (2.6.11) box which has 1Gig
RAM and
> > an additional 1Gig swap space. The server handles a lot of people
downloading
> > sermons from our church website (which are generally about 16-18Meg MP3
> > files), but I can't figure out how to keep the server from running out
of
> > memory.
>
> As I'm still running 2.4 kernels mainly, I don't recall if the 2.6 series
has
> the experimental "khttpd" option on it. If it does, perhaps you should
try
> that since your MPEG3 files are not dynamically generated but are subject
to
> the operation of "copy file to network unaltered."
>
> Maybe you'll test this by downgrading back to 2.4 temporarily (if you have
to).

zen wrote:
> Will using a CacheFile and zlib (php & zlib.output_compression) not prevent

No. The problem is byteranges mishandling, not file size, and is simply
fixed by unsetting all range-related headers (there's also a patch
somewhere on Joe Orton's page at http://people.apache.org/~jorton/ ).
Compressing by any means (the obvious one being mod_deflate of course)
is irrelevant, and PHP would just make things far worse.

--
Nick Kew

Just wanted to follow up with the solution to this. Thanks to everyone
for their suggestion. Nick, your theory was right - it was indeed
related to the byterange bug in the 2.0.x branch of Apache (apparently
it's fixed in the 2.1.x branch).

It seems to only occur if:
- the server admin has setup a CGI script to handle the transfer of
files being downloaded, and
- the client is using a download accelerator program that splits the
transfer into multiple requests for faster downloading.

Here's Joe Orton's patch for the 2.0.x branch of Apache:

http://people.apache.org/~jorton/ap_byterange.diff

and here's my message to the Apache Users mailing list where I show the
before-and-after effects of patching my 2.0.54 server:

---------------------------------------------------------------

Joe, I just wanted to thank you again. The byterange patch you gave me
worked just beautifully.

Once I understood what the problem was, I was able to test it more
thoroughly. I took a copy of Star Downloader and configured it to split
up a single file into 10 chunks for faster downloading. Then I took a
freshly-started Apache 2.0.54 server and told Star Downloader to being
downloading a single file. It immediately broke it into several chunks
and made 10 requests of the Apache server, each with a different byterange.

The effects on my server were dramatic, as you can see from this "vmstat
1" (every line represents one second)

procs -----------memory---------- ---swap-- --system-- ----cpu----
2 0 128368 728244 12044 105432 0 0 1358 9823 6 14 50 29
0 0 128368 626192 12048 108288 0 0 1303 13096 8 19 63 9
10 0 128368 470044 12048 108288 0 0 1283 49044 18 38 45 0
12 0 128368 169204 12048 108548 0 0 1228 106594 29 71 0 0
8 4 127664 12260 5496 71652 0 204 1298 73347 22 69 0 8
0 17 178472 11996 5544 37556 32 51780 1380 7073 8 43 0 49
0 16 179200 12368 5564 38704 32 1712 1319 1277 1 3 0 96
0 14 180056 12244 5612 41284 568 968 1332 4709 2 4 0 94
0 14 180056 12120 5636 41884 696 1484 1394 1692 1 4 0 96
0 14 182116 11988 5652 43704 176 4772 1488 2841 2 5 0 93
0 16 198168 12236 5676 44920 128 17276 1466 873 0 4 0 95
0 17 218620 12416 5680 45568 260 22432 1455 694 1 5 0 95
0 19 221760 12108 5688 47040 224 5560 1396 3476 2 6 0 92
0 18 224692 11988 5724 48012 612 3744 1403 1796 0 3 0 96
0 20 233944 11980 5752 48544 0 10452 1355 811 1 3 0 97
0 17 234612 12096 5808 50344 204 1456 1356 2878 2 4 0 94
0 16 234876 12096 5868 51816 300 1524 1377 1578 1 3 0 97
0 19 271892 12092 5896 51888 384 39996 1518 2744 2 11 0 88
0 17 314812 12156 5920 51572 752 43876 1467 959 0 9 0 90

Within 5 seconds, I had dropped from 728,244b free RAM to only 12,260b
free. Immediately after that and the swap space usage began to rise at
an alarming rate. All that just from a SINGLE download request split up
into 10 smaller requests! No wonder the server was choking so often.

I then restarted the server and tested the same process with Star
Downloader, but this time requesting the file directly rather than going
through a download CGI script. This time Apache didn't blink an eye -
it handled all 10 split-up requests with barely a flicker in memory
usage. So your theory was right - the problem only occurred when a file
was requested via a download accelerator and only when an intermediary
CGI was handling the file transfer on the server side.

Next, I applied the patch, rebuilt and restarted Apache, and made sure
Star Downloader could still download the file directly (without the CGI
download script). Everything was still fine - the request was split
into 10 requests and Apache handled them all easily.

Finally, the big test. I again requested the file with Star Downloader
using the server's CGI download script. This time, though, the request
was not split into 10 smaller requests. Instead, Star Downloader had to
content itself with a single downloading process, and Apache's memory
usage held steady.

So, hooray! Thanks again, and if you have any influence to get that
script backported to the Apache 2.0.x branch, that'd be wonderful.