loading huge number of rules in iptables (blocklist)

Hello,
I am playing round with the blocklist file obtained from peerguardian
(level1.gz). I have written a bash function which I call in my iptables
script to load the rules to block the ipranges given in the block list
file. The file has the a range of IPs to blocked listed no each line.

The following function actually loads the gzipped file
(e.g. /etc/firewall/level1.gz, defined by the P2PBLOCKLISTFILE variable)
-------------------------------------------------------------------------
#function that creats the rules to block the traffic from blocked list
function FuncBlockedIPsRules {
echo "Making rules for Blocked IPs traffic";
while read Line; do
#get the ip address range from the file
IpRange=`echo -n $Line | sed -e 's/.*:\(.*\)-\(.*\)/\1-\2/'`;
#drop the traffic from this port range
$IPTABLES -A ${CHN_BTBLOCKEDIPS} \
-m iprange --src-range $IpRange -j DROP
done < <(zcat ${P2PBLOCKLISTFILE} | iconv -f latin1 -t utf-8 - |
dos2unix)
}
-------------------------------------------------------------------------


Now, currently, there are around 151,000 ipranges listed in level1.gz to
block. So the above function's loop goes over these many times inserting
the rules for each range. And this is taking huge amount of time: in
over 50 minutes, only around 10% rules have been loaded on my router
running Etch (Pentium III, 449MHz, 380 MB RAM).

How can I speed this up? Advice? I am sure I am not doing this in a
smart way, this is at best a brute force method. But this is my first
try and I am sure better methods exist.

thanks,
->HS

--
(Remove all caps,if any, from my email address to get the correct one.
Apologies for the inconvenience but this is to reduce spam.)

H.S. wrote:
> Hello,
> I am playing round with the blocklist file obtained from peerguardian
> (level1.gz). I have written a bash function which I call in my iptables
> script to load the rules to block the ipranges given in the block list
> file. The file has the a range of IPs to blocked listed no each line.
>
> The following function actually loads the gzipped file
> (e.g. /etc/firewall/level1.gz, defined by the P2PBLOCKLISTFILE variable)
> -------------------------------------------------------------------------
> #function that creats the rules to block the traffic from blocked list
> function FuncBlockedIPsRules {
> echo "Making rules for Blocked IPs traffic";
> while read Line; do
> #get the ip address range from the file
> IpRange=`echo -n $Line | sed -e 's/.*:\(.*\)-\(.*\)/\1-\2/'`;
> #drop the traffic from this port range
> $IPTABLES -A ${CHN_BTBLOCKEDIPS} \
> -m iprange --src-range $IpRange -j DROP
> done < <(zcat ${P2PBLOCKLISTFILE} | iconv -f latin1 -t utf-8 - |
> dos2unix)
> }
> -------------------------------------------------------------------------
>
>
> Now, currently, there are around 151,000 ipranges listed in level1.gz to
> block. So the above function's loop goes over these many times inserting
> the rules for each range. And this is taking huge amount of time: in
> over 50 minutes, only around 10% rules have been loaded on my router
> running Etch (Pentium III, 449MHz, 380 MB RAM).
>
> How can I speed this up? Advice? I am sure I am not doing this in a
> smart way, this is at best a brute force method. But this is my first
> try and I am sure better methods exist.
>
> thanks,
> ->HS
>

If you have large numbers of ip addresses, and want to apply the same
rules to those addresses or ranges, then ipsets may be a better method.
I haven't tried it myself as yet, but in my research for planning a
new firewall and router, they looked like a much more efficient way to
do exactly this sort of thing. You can also add or remove addresses to
an ipset without changing your iptables rules - very useful if the
iptables are generated by a script such as shorewall (it was via the
shorewall website that I first read about ipsets).

On 2007-03-21, David Brown <david.brown@hesbynett.removethisbit.no> wrote:
>
> If you have large numbers of ip addresses, and want to apply the same
> rules to those addresses or ranges, then ipsets may be a better method.
> I haven't tried it myself as yet, but in my research for planning a
> new firewall and router, they looked like a much more efficient way to
> do exactly this sort of thing. You can also add or remove addresses to
> an ipset without changing your iptables rules - very useful if the
> iptables are generated by a script such as shorewall (it was via the
> shorewall website that I first read about ipsets).

Thanks for the suggestions. I have had this one from at least two other
sources. I appears to hold some promise. The other suggestion I have had
to look into using nf-hipac.

regards,
->HS

--
(Remove all caps,if any, from my email address to get the correct one.
Apologies for the inconvenience but this is to reduce spam.)