[squid-users] Squid memory leak?

I've been chasing what looks like a memory leak for quite some time. =
I've seen it all through
the Squid 2.X releases, and it still is present in 2.5STABLE4. I've seen =
it on machines running
HP-UX 10.20 and on Red Hat Linux. Currently I'm running on three =
identical Dell 2650 servers,
each with dual 1.8 GHz processors, 2 GB of RAM, and two 20 GB cache =
partitions. They are all
running Red Hat ES 2.1, kernel 2.4.9-e.25smp.

Squid is built with these configuration options:
Squid Cache: Version 2.5.STABLE4
configure options: --prefix=3D/local/proxy/squid =
--enable-cache-digests
--enable-underscores --with-pthreads --enable-storeio=3Daufs,ufs
--enable-removal-policies=3Dheap --enable-gnuregex

With a full cache, Squid memory utilization starts at about 250 MB. It =
then grows at about
300 MB per week. The graph at
ftp://ftp.sas.com/pub/nam/squid-mem.png
shows how the memory grows.
We see about 80 requests per second during business hours. The graph
ftp://ftp.sas.com/pub/nam/squid-reqs.png
shows our requests per second.
We use DNS round-robin to load balance requests between the three =
systems.

Each squid process forwards most requests on to a Trend Interscan =
VirusWall process listening
on port 8080 on the same server. It also sends the URLs through the =
"adzap" redirector. ACLs
are used to decide which URLs are sent through the redirector and also =
which URLs are
sent to the virus scanner.

Here's some squid information output from the cachemgr.cgi:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D
Squid Object Cache: Version 2.5.STABLE4
Start Time: Sat, 20 Sep 2003 19:29:16 GMT
Current Time: Mon, 29 Sep 2003 21:14:57 GMT
Connection information for squid:
Number of clients accessing cache: 0
Number of HTTP requests received: 17264989
Number of ICP messages received: 1961488
Number of ICP messages sent: 1961585
Number of queued ICP replies: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 1321.4
Average ICP messages per minute since start: 300.3
Select loop called: 90171815 times, 8.694 ms avg
Cache information for squid:
Request Hit Ratios: 5min: 62.3%, 60min: 65.9%
Byte Hit Ratios: 5min: 15.8%, 60min: 21.1%
Request Memory Hit Ratios: 5min: 4.5%, 60min: 4.2%
Request Disk Hit Ratios: 5min: 18.4%, 60min: 12.5%
Storage Swap size: 31876696 KB
Storage Mem size: 98348 KB
Mean Object Size: 14.58 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.02317 0.02317
Cache Misses: 0.16775 0.16775
Cache Hits: 0.01309 0.01469
Near Hits: 0.09736 0.10857
Not-Modified Replies: 0.01387 0.01469
DNS Lookups: 0.00464 0.00464
ICP Queries: 0.00316 0.00541
Resource usage for squid:
UP Time: 783941.680 seconds
CPU Time: 118154.720 seconds
CPU Usage: 15.07%
CPU Usage, 5 minute avg: 65.24%
CPU Usage, 60 minute avg: 60.68%
Process Data Segment Size via sbrk(): 556455 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 33606
Memory usage for squid via mallinfo():
Total space in arena: 556455 KB
Ordinary blocks: 491460 KB 532257 blks
Small blocks: 0 KB 0 blks
Holding blocks: 19384 KB 12 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 64995 KB
Total in use: 510844 KB 89%
Total free: 64995 KB 11%
Total size: 575839 KB
Memory accounted for:
Total accounted: 263830 KB
memPoolAlloc calls: 2122989169
memPoolFree calls: 2118393830
File descriptor usage for squid:
Maximum number of file descriptors: 4096
Largest file desc currently in use: 877
Number of file desc currently in use: 465
Files queued for open: 0
Available number of file descriptors: 3631
Reserved number of file descriptors: 100
Store Disk files open: 4
Internal Data Structures:
2199786 StoreEntries
13566 StoreEntries with MemObjects
13530 Hot Object Cache Items
2186456 on-disk objects
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
What is most interesting from this is that mallinfo says 556455 KB is in =
the arena, but
squid can only account for 263830 KB of it.=20

Here's the interesting bits of our configuration:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
http_port 80 8000
cache_peer 127.0.0.1 parent 8080 7 no-query no-digest default
cache_peer inetgw02.unx.sas.com sibling 80 3130 proxy-only
cache_peer inetgw03.unx.sas.com sibling 80 3130 proxy-only
cache_peer inetgw02.unx.sas.com parent 8080 7 no-query no-digest =
round-robin
cache_peer inetgw03.unx.sas.com parent 8080 7 no-query no-digest =
round-robin
cache_mem 96 MB
cache_swap_low 95
cache_swap_high 97
maximum_object_size 700 MB
maximum_object_size_in_memory 32 KB
cache_replacement_policy heap LFUDA
memory_replacement_policy heap GDSF
cache_dir aufs /cache/cache1/squid 16384 53 254
cache_dir aufs /cache/cache2/squid 16384 53 254
cache_store_log none
emulate_httpd_log on
acl all src 0.0.0.0/0.0.0.0
acl noAds myport 80
acl Reads method GET HEAD
acl CIDR_A src 10.0.0.0/255.0.0.0
acl virus_proto proto HTTP
acl novirus-url urlpath_regex -i \.gif(\?.*)?$ \.jpg(\?.*)?$ =
\.png(\?.*)?$
ident_lookup_access allow CIDR_A
ident_lookup_access deny all
memory_pools on
log_icp_queries off
client_db off
always_direct allow novirus-url
always_direct allow !virus_proto
never_direct allow all
strip_query_terms off
redirect_program /opt/adzap/wrapzap
redirect_children 10
redirector_access deny !Reads
redirector_access allow noAds
redirector_bypass on
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D

Here's some messages that are showing up in the cache.log file.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
2003/09/29 16:38:26| sslReadServer: FD 578: read failure: (104) =
Connection reset by peer
2003/09/29 16:39:00| sslReadServer: FD 574: read failure: (104) =
Connection reset by peer
2003/09/29 16:41:44| sslReadServer: FD 560: read failure: (104) =
Connection reset by peer
2003/09/29 16:42:14| sslReadServer: FD 485: read failure: (104) =
Connection reset by peer
2003/09/29 16:42:54| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:42:55| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:42:59| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:03| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:07| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:08| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:17| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:21| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:31| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:32| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:33| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:34| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:46| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:43:47| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:44:00| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
2003/09/29 16:44:00| urlParse: Illegal character in hostname =
'inetgw01.unx.sas.com:80inetgw01.unx.sas.com'
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3 D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
Does anyone have an idea of why I'm consuming so much memory? Am I just =
running into memory
fragmentation issues? Should I be linking with a different version of =
malloc? Would the
configuration option '--enable-dlmalloc' help?
I've worked around this problem by having a cron job check how large the =
squid process is.
If it grows too large I restart it. Currently it gets restarted twice a =
month.
-------------------------------------------------------------------------=
-----
Mike Mitchell
Mike.Mitchell@sas.com
(919) 677-8000 X16793