Large binary newreader help please!

I am running redhat 9 on a small LAN, redhat machine is the
gateway/firewall/server for this small system. I have ADSL with fixed IP,
3Mbps download and 380Kbps upload. I want to open up a large binary newsgroup
to search for some files and I always do this with xnews on my XP machine
that is on the LAN. I open the newsgroup and download all of the headers. The
xnews program sorts and threads all of the headers and I am presented with a
list of posts that are threaded into little folder icons to hold the threads,
and binary files are grouped into little "Rubik's Cube" icons, four
grouped squares indicate that the binary file is complete with all parts
present. I can queue the binary files that I want by highlighting them and
then pressing the spacebar. F4 will allow me to download them and browse for
a folder to put them in. I use PAN in linux xwindows to do pretty much the
same thing. All well and good...

Here is the problem. I subscribe to a high speed news server (Giganews) with
a very high retention rate and it is almost 100% complete. When I enter the
binary newsgroup that I would like to go in, the newsgroup has about 2.5
million headers in it. To download all of the headers in this very large
newsgroup takes sometimes a couple of hours. By the time I get them all, all
of my system RAM is completely used up (1Gb DDR RAM) and I am about 1.2Gb
into the windows swap file. After downloading all of the headers, xnews will
sort and thread all of the articles. Because I am so far down into the swap
file by this time, the sorting and threading can take over an hour. So, over
three hours later, I am finally able to begin searching the newsgroup.

The last time I did this, I had to go to bed because it was taking so long
and when I woke up, all headers were downloaded, threaded, and sorted, but
the windows machine was slowed to a bloody crawl. Any action would take
forever as we were 1.2Gb into the swap file by now. As I was getting ready to
pick out the files I wanted, the power guy comes to the house to trim the
trees and he tells me that they have to cut the power for a while. Before I
could save or do anything, the power goes off and everything is lost. <sob!>

I really want to get these binary file but do not want to tie up my windows
machine like this. Is there any kind of program that I could use in linux to
get the headers, sort them, group them and show which are complete to
download for the console? I think that PAN would choke up too on this very
large group, I would prefer a console program instead. I have slrn installed
and I guess I could change my NNTPSERVER export to make it use the high speed
servers. I have this in my /etc/profle to give export my NNTPSERVER:


NNTPSERVER='newsgroups.bellsouth.net' && export NNTPSERVER

My high speed server requires a login and password, should that also be put
in the /etc/profle file (If so, how?) or will slrn ask me for user and pass
when it connects?

Is slrn up to this monumental task or is there anything better for this?

Redhat server has 1.5Gb of SDRAM and linux swap on /dev/hda3 is 1,279Mb in
size. I use the linux box as a small family web and FTP server and hopefully
will not bring down the system with this newsgroup task. I want the linux
machine to do this task so that I am free to use the XP machine to do other
things while this is going on.

Can anyone please give me some tips or suggestions on how to go about doing
this safely?

Thanks.

--
~Ohmster

On Sun, 02 May 2004 16:25:41 GMT, Ohmster wrote:

> NNTPSERVER='newsgroups.bellsouth.net' && export NNTPSERVER
>
> My high speed server requires a login and password, should that also be put
> in the /etc/profle file (If so, how?) or will slrn ask me for user and pass
> when it connects?


Example line in ~/.slrnrc
nnrpaccess news.comcast.giganews.com news_login_here news_password_here

Bit Twister <BitTwister@localhost.localdomain> wrote in
news:slrnc9a9c0.n2f.BitTwister@wb.home.invalid:

>> NNTPSERVER='newsgroups.bellsouth.net' && export NNTPSERVER
>>
>> My high speed server requires a login and password, should that also be
>> put in the /etc/profle file (If so, how?) or will slrn ask me for user
>> and pass when it connects?
>
>
> Example line in ~/.slrnrc
> nnrpaccess news.comcast.giganews.com news_login_here news_password_here

So if my user was "user" and password was "password", the line would look
like this"

nnrpaccess news.comcast.giganews.com user password

Is "nnrpaccess" correct or is that a typo? (nntpaccess?)

Thanks Bit Twister.

--
~Ohmster

Ohmster wrote:

> Here is the problem. I subscribe to a high speed news server (Giganews)
> with a very high retention rate and it is almost 100% complete. When I
> enter the binary newsgroup that I would like to go in, the newsgroup has
> about 2.5 million headers in it. To download all of the headers in this
> very large newsgroup takes sometimes a couple of hours. By the time I get
> them all, all of my system RAM is completely used up (1Gb DDR RAM) and I
> am about 1.2Gb into the windows swap file. After downloading all of the
> headers, xnews will sort and thread all of the articles. Because I am so
> far down into the swap file by this time, the sorting and threading can
> take over an hour. So, over three hours later, I am finally able to begin
> searching the newsgroup.

O.K. here is what I do in Pan. First I click to select the group in the
subscribed groups pane. Then I right click on the group and select more
download options from the context menu.

In the dialog box that appears I click on the up arrow of the download
recent headers option until it reads several thousand then I hit the
execute button.

I wait a minute or two at most for the most recent several thousand headers
to arrive.

Styvaen.

Bit Twister <BitTwister@localhost.localdomain> wrote in
news:slrnc9a9c0.n2f.BitTwister@wb.home.invalid:

> Example line in ~/.slrnrc
> nnrpaccess news.comcast.giganews.com news_login_here news_password_here

Bit Twister, is binary download support decent in slrn? I have never used it
for binary downloading. Are the parts grouped in slrn? e.g.:

file part 1/56
file part 2/56
file part 3/56

etc.?

Do you know the commands for marking and downloading multipart binary posts
in slrn off the top of your head?

Thanks.

--
~Ohmster

On Sun, 02 May 2004 16:51:10 GMT, Ohmster wrote:
>>
>> Example line in ~/.slrnrc
>> nnrpaccess news.comcast.giganews.com news_login_here news_password_here
>
> So if my user was "user" and password was "password", the line would look
> like this"
>
> nnrpaccess news.comcast.giganews.com user password

If _user_ is your news server login id on the news server
_news.comcast.giganews.com_ then yup, looks good. :)

> Is "nnrpaccess" correct or is that a typo? (nntpaccess?)

That was a cut/paste from my /.slrnrc with *_here changed.

Hey when in doubt,
locate slrn
and grep a file like /usr/share/doc/slrn-0.9.8.0/slrn.rc

On Sun, 02 May 2004 16:59:31 GMT, Ohmster wrote:

> Bit Twister, is binary download support decent in slrn? I have never used it
> for binary downloading. Are the parts grouped in slrn? e.g.:

Cannot say, I use Thunderbird for binary viewing, slrn for text reading.

> Do you know the commands for marking and downloading multipart binary posts
> in slrn off the top of your head?

Nope, you could look through the docs you found in my other reply.

As I read your original post, I thought slrnpull would be
consideration for off hour download.

Bit Twister <BitTwister@localhost.localdomain> wrote in
news:slrnc9aes1.n9h.BitTwister@wb.home.invalid:

> If _user_ is your news server login id on the news server
> _news.comcast.giganews.com_ then yup, looks good. :)
>
>> Is "nnrpaccess" correct or is that a typo? (nntpaccess?)
>
> That was a cut/paste from my /.slrnrc with *_here changed.
>
> Hey when in doubt,
> locate slrn
> and grep a file like /usr/share/doc/slrn-0.9.8.0/slrn.rc

Bit Twister,

I edited my ~/.slrnrc file (Funny, it was named .slrn.rc, I copied it over to
..slrnrc.) file as you said and put in my giganews NNTP server info. When I
started up slrn, I still ended up going to my bellsouth NNTP server anyway.
Could not find any way to override the default NNTP server export in the
/etc/profile file so I edited that one to export the giganews server. I
really would have liked to leave the main NNTP server set to bellsouth and
then just use the giganews server as needed but could not find a way to do
this.

Anyway, am now at 500K out of 2.5M headers, will see if this can get the job
done. Not sure what binary files are going to look like in slrn, I sure hope
that they are threaded and grouped appropriatly and that marking them for
decodeing won't be too difficult of a process.

Will let you know how it goes. Thanks for your help.

--
~Ohmster

On Sun, 02 May 2004 18:58:07 GMT, Ohmster wrote:

> put in my giganews NNTP server info. When I
> started up slrn, I still ended up going to my bellsouth NNTP server anyway.
> Could not find any way to override the default NNTP server export in the
> /etc/profile file so I edited that one to export the giganews server. I
> really would have liked to leave the main NNTP server set to bellsouth and
> then just use the giganews server as needed but could not find a way to do
> this.

You realy aught to try the command man slrn

My solution.
I created a script and pass which server I want to access.
I then have a launch icon for each server with a command line like

/site/bin/news comcast
/site/bin/news giga


news script snippet follow:

case $1 in
giga) NNTPSERVER=news.comcast.giganews.com
;;

comcast)
NNTPSERVER=netnews.comcast.net
# NNTPSERVER=63.240.76.16 # New Jersey
# NNTPSERVER=204.127.199.17 # San Francisco
NNTPSERVER=204.127.204.17 # Central usa
# NNTPSERVER=216.148.227.77 # San Francisco
;;

*) echo " "
echo "$1 is an invalid selection"
echo " "
echo "Usage: news giga"
echo " news comcast"
exit 1
;;
esac

export NNTPSERVER

xterm -tn xterm -fn 10x20 -title "news $1 " -geom 100x43+200+200 -e slrn -C

Bit Twister <BitTwister@localhost.localdomain> wrote in
news:slrnc9ai5i.ne4.BitTwister@wb.home.invalid:

OMG, this is so freaking kewl, you are a genious, Bit Twister!


> You realy aught to try the command man slrn

Did that already, I couldn't find a way to lauch slrn from the console and
specify which server that I want to use easily. Besides, I would *never* have
been able to figre all of this out. :)

>
> My solution.
> I created a script and pass which server I want to access.
> I then have a launch icon for each server with a command line like
>
> /site/bin/news comcast
> /site/bin/news giga


So what is the name that you created for this script? "news"?
You put the script in /site/bin/ ? Can I just put it in ~/scripts ?

Your script will accept a command line argument like "comcast" or "giga"?

Oh man I gotta try this....

In my ~/.slrnrc I can put my username and password for the server and it
works when I use that server in my /etc/profile file. How do you handle the
user/pass issue with your script? Do you just answer these questions when
slrn loads? This is a bit difficult because giganews gave me a real weird
username and password, I will never be able to remember these items. Can your
script be modified to accept user/pass strings or can your script be modified
to read them from a file?


>
> news script snippet follow:
>
> case $1 in
> giga) NNTPSERVER=news.comcast.giganews.com
> ;;
>
> comcast)
> NNTPSERVER=netnews.comcast.net
> # NNTPSERVER=63.240.76.16 # New Jersey
> # NNTPSERVER=204.127.199.17 # San Francisco
> NNTPSERVER=204.127.204.17 # Central usa
> # NNTPSERVER=216.148.227.77 # San Francisco
> ;;
>
> *) echo " "
> echo "$1 is an invalid selection"
> echo " "
> echo "Usage: news giga"
> echo " news comcast"
> exit 1
> ;;
> esac
>
> export NNTPSERVER
>
> xterm -tn xterm -fn 10x20 -title "news $1 " -geom 100x43+200+200 -e
> slrn -C

This last line I presume sets the window geometry when you use slrn in
xwindows. What does "slrn -C" do?

Wow, this is really cool. Thanks a lot Bit Twister!


--
~Ohmster