[squid-users] squid tries to rebuild cache but fails...

Hi,

I have here a Netra-machine with Solaris 8 and 1GB RAM. This machine is=

serving as mailhost/mailscan-machine and Webproxy. Today,squid stopped
because it claimed not to have enough storage space to store and it tri=
ed
to rebuild the store database. But even when there is enough storage sp=
ace
and inodes available, it still crashes:

[..]
root@host# tail -f /opt/sfw/squid/var/logs/cache.log

2004/04/27 12:30:59| Starting Squid Cache version 2.5.STABLE2-20030411 =
for
sparc-sun-solaris2.9...
2004/04/27 12:30:59| Process ID 11178
2004/04/27 12:30:59| With 1024 file descriptors available
2004/04/27 12:30:59| DNS Socket created at 0.0.0.0, port 59790, FD 5
2004/04/27 12:30:59| Adding nameserver 212.121.128.10 from /etc/resolv.=
conf
2004/04/27 12:30:59| Adding nameserver 212.121.128.11 from /etc/resolv.=
conf
2004/04/27 12:30:59| Unlinkd pipe opened on FD 10
2004/04/27 12:30:59| Swap maxSize 18432000 KB, estimated 1417846 object=
s
2004/04/27 12:30:59| Target number of buckets: 70892
2004/04/27 12:30:59| Using 131072 Store buckets
2004/04/27 12:30:59| Max Mem size: 131072 KB
2004/04/27 12:30:59| Max Swap size: 18432000 KB
2004/04/27 12:30:59| Store logging disabled
2004/04/27 12:30:59| Rebuilding storage in /cache (DIRTY)
2004/04/27 12:30:59| Using Least Load store dir selection
2004/04/27 12:30:59| Set Current Directory to /opt/sfw/squid/var/logs
2004/04/27 12:30:59| Loaded Icons.
2004/04/27 12:30:59| Accepting HTTP connections at 0.0.0.0, port 8080, =
FD
11.
2004/04/27 12:30:59| Accepting ICP messages at 0.0.0.0, port 3130, FD 1=
2.
2004/04/27 12:30:59| WCCP Disabled.
2004/04/27 12:30:59| Ready to serve requests.
2004/04/27 12:31:00| Store rebuilding is 0.2% complete
2004/04/27 12:31:15| Store rebuilding is 17.4% complete
2004/04/27 12:31:32| Store rebuilding is 22.0% complete
2004/04/27 12:31:48| Store rebuilding is 23.2% complete
2004/04/27 12:32:03| Store rebuilding is 38.1% complete
Apr 27 12:32:16 boe-stvw-p2 squid[11178]: [ID 702911 local4.alert] Writ=
e
failure -- check your disk space and cache.log
2004/04/27 12:32:16| diskHandleWrite: FD 8: disk write error: (28) No s=
pace
left on device
FATAL: Write failure -- check your disk space and cache.log
Squid Cache (Version 2.5.STABLE2-20030411): Terminated abnormally.
CPU Usage: 38.110 seconds =3D 20.550 user + 17.560 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
2004/04/27 12:32:19| Starting Squid Cache version 2.5.STABLE2-20030411 =
for
sparc-sun-solaris2.9...
2004/04/27 12:32:19| Process ID 11194
2004/04/27 12:32:19| With 1024 file descriptors available
2004/04/27 12:32:19| DNS Socket created at 0.0.0.0, port 59791, FD 5
2004/04/27 12:32:19| Adding nameserver 212.121.128.10 from /etc/resolv.=
conf
2004/04/27 12:32:19| Adding nameserver 212.121.128.11 from /etc/resolv.=
conf
2004/04/27 12:32:19| Unlinkd pipe opened on FD 10
2004/04/27 12:32:19| Swap maxSize 18432000 KB, estimated 1417846 object=
s
2004/04/27 12:32:19| Target number of buckets: 70892
2004/04/27 12:32:19| Using 131072 Store buckets
2004/04/27 12:32:19| Max Mem size: 131072 KB
2004/04/27 12:32:19| Max Swap size: 18432000 KB
2004/04/27 12:32:19| Store logging disabled
2004/04/27 12:32:19| Rebuilding storage in /cache (DIRTY)
2004/04/27 12:32:19| Using Least Load store dir selection
2004/04/27 12:32:19| Set Current Directory to /opt/sfw/squid/var/logs
2004/04/27 12:32:19| Loaded Icons.
2004/04/27 12:32:19| Accepting HTTP connections at 0.0.0.0, port 8080, =
FD
11.
2004/04/27 12:32:19| Accepting ICP messages at 0.0.0.0, port 3130, FD 1=
2.
2004/04/27 12:32:19| WCCP Disabled.
2004/04/27 12:32:19| Ready to serve requests.
2004/04/27 12:32:20| Store rebuilding is 0.2% complete
2004/04/27 12:32:35| Store rebuilding is 19.1% complete
2004/04/27 12:32:39| Preparing for shutdown after 109 requests
2004/04/27 12:32:39| Waiting 30 seconds for active connections to finis=
h
2004/04/27 12:32:39| FD 11 Closing HTTP connection
2004/04/27 12:32:50| Store rebuilding is 36.2% complete
Apr 27 12:32:56 boe-stvw-p2 squid[11194]: [ID 702911 local4.alert] Writ=
e
failure -- check your disk space and cache.log
2004/04/27 12:32:56| diskHandleWrite: FD 8: disk write error: (28) No s=
pace
left on device
FATAL: Write failure -- check your disk space and cache.log
Squid Cache (Version 2.5.STABLE2-20030411): Terminated abnormally.
CPU Usage: 33.610 seconds =3D 18.620 user + 14.990 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
2004/04/27 12:33:27| Starting Squid Cache version 2.5.STABLE2-20030411 =
for
sparc-sun-solaris2.9...
2004/04/27 12:33:27| Process ID 11239
2004/04/27 12:33:27| With 1024 file descriptors available
2004/04/27 12:33:27| DNS Socket created at 0.0.0.0, port 59792, FD 5
2004/04/27 12:33:27| Adding nameserver 212.121.128.10 from /etc/resolv.=
conf
2004/04/27 12:33:27| Adding nameserver 212.121.128.11 from /etc/resolv.=
conf
2004/04/27 12:33:27| Unlinkd pipe opened on FD 10
2004/04/27 12:33:27| Swap maxSize 18432000 KB, estimated 1417846 object=
s
2004/04/27 12:33:27| Target number of buckets: 70892
2004/04/27 12:33:27| Using 131072 Store buckets
2004/04/27 12:33:27| Max Mem size: 131072 KB
2004/04/27 12:33:27| Max Swap size: 18432000 KB
2004/04/27 12:33:27| Store logging disabled
2004/04/27 12:33:27| Rebuilding storage in /cache (DIRTY)
2004/04/27 12:33:27| Using Least Load store dir selection
2004/04/27 12:33:27| Set Current Directory to /opt/sfw/squid/var/logs
2004/04/27 12:33:27| Loaded Icons.
2004/04/27 12:33:27| Accepting HTTP connections at 0.0.0.0, port 8080, =
FD
11.
2004/04/27 12:33:27| Accepting ICP messages at 0.0.0.0, port 3130, FD 1=
2.
2004/04/27 12:33:27| WCCP Disabled.
[..]

as you can see, it tries to restart everytime.


[..]
root@host# uname -a
SunOS boe-stvw-p2 5.8 Generic_108528-27 sun4u sparc SUNW,UltraAX-i2

root@host# df -o i
df: operation not applicable for FSType fd
df: operation not applicable for FSType mntfs
df: operation not applicable for FSType nfs
df: operation not applicable for FSType proc
df: operation not applicable for FSType tmpfs
Filesystem iused ifree %iused Mounted on
/dev/md/dsk/d10 4568 367144 1% /
/dev/md/dsk/d30 45367 466633 9% /usr
/dev/md/dsk/d40 19357 460643 4% /var
/dev/md/dsk/d60 1592287 935713 63% /cache
/dev/md/dsk/d50 25493 318571 7% /opt


root@host# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d10 1487071 1421933 5656 100% /
/dev/md/dsk/d30 4130982 1009143 3080530 25% /usr
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
/dev/md/dsk/d40 3916782 476513 3401102 13% /var
swap 6841480 24 6841456 1% /var/run
swap 6856384 14928 6841456 1% /tmp
/dev/md/dsk/d60 20587657 16174440 4207341 80% /cache
/dev/md/dsk/d50 2055463 1147450 846350 58% /opt
[..]

as you can see, there is nearly no space anymore on /. This is already
fixed (more than 160Mbyte now free), but squid still crashes at the sam=
e
moment (store rebuilding).

I have here the squid.conf with all comments and emtpy lines taken out:=


[..]
root@host# grep -v ^# squid.conf|more
http_port 8080
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
cache_mem 128 MB
cache_swap_low 90
cache_swap_high 95
maximum_object_size 8192 KB
minimum_object_size 0 KB
maximum_object_size_in_memory 32 KB
ipcache_size 2048
ipcache_low 90
ipcache_high 95
fqdncache_size 2048
cache_replacement_policy lru
memory_replacement_policy lru
cache_dir ufs /cache 18000 32 256
cache_access_log /opt/sfw/squid/var/logs/access.log
cache_access_log /opt/sfw/squid/var/logs/access.log
cache_log /opt/sfw/squid/var/logs/cache.log
cache_store_log none
pid_filename /opt/sfw/squid/var/logs/squid.pid
debug_options ALL,1
auth_param basic children 5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern . 60 40% 43200
quick_abort_pct 95
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow all
http_access deny all
http_reply_access allow all
icp_access allow all
logfile_rotate 10
coredump_dir /opt/sfw/squid/var/logs
ie_refresh on
[..]


I first thought that squid tries to write somewhere on the root-Filesys=
tem
and there is not enough space. Unfortunately, I get the same error even=

when I freed some space.

Can you give me some hints where to check for enough space or to see wh=
at
really is wrong here?

Mit freundlichen Gr=FC=DFen,

Hanno Wagner
--
Hanno Wagner - Unix-C/S-Systemadministrator
Boerse Stuttgart AG
Email: Hanno.Wagner@boerse-stuttgart.de Tel: +49-711-222 985-569

Diese E-Mail enth=E4lt vertrauliche oder rechtlich gesch=FCtzte Informa=
tionen.
Wenn Sie nicht der beabsichtigte Empf=E4nger sind, informieren Sie bitt=
e
sofort den Absender und l=F6schen Sie diese E-Mail. Das unbefugte Kopie=
ren
dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Information=
en
ist nicht gestattet.

The information contained in this message is confidential or protected =
by
law. If you are not the intended recipient, please contact the sender a=
nd
delete this message. Any unauthorised copying of this message or
unauthorised distribution of the information contained herein is
prohibited.
=