Large binary newreader help please!

Bit Twister <BitTwister@localhost.localdomain> wrote in
news:slrnc9af7e.n9h.BitTwister@wb.home.invalid:

> As I read your original post, I thought slrnpull would be
> consideration for off hour download.

Doing it now with slrn. OMG, this is brutal. Might have to go back to Windows
for this job...

I really need a *good* binary news puller for this. Will have to search
around a bit. Thanks Bit Twister.

--
~Ohmster

On Sun, 02 May 2004 19:29:48 GMT, Ohmster wrote:
> Bit Twister <BitTwister@localhost.localdomain> wrote in
> news:slrnc9ai5i.ne4.BitTwister@wb.home.invalid:
>
> OMG, this is so freaking kewl, you are a genious, Bit Twister!
>
>
>> You realy aught to try the command man slrn
>
> Did that already, I couldn't find a way to lauch slrn from the
> console and specify which server that I want to use easily.

Hmmm, me thinks _easily_ is your problem. :)

Looking at the man snippet
SYNOPSIS
slrn [-aCdknmw] [-C-] [-Dname] [-f newsrc-file] [-i config-file] [-k0]
[--create] [-create] [--debug file] [--help] [--inews] [--kill-log
file] [--nntp [-h server] [-p port]] [--spool] [--version]

I would have guessed
slrn --nntp -h which_server_here
would have worked.

>
> So what is the name that you created for this script? "news"?

Yup

> You put the script in /site/bin/ ?

Yup

> Can I just put it in ~/scripts ?

I'll guess yes. I'll assume $HOME/scripts is in your $PATH and you do
a chmod +x news after you create the script.

> Your script will accept a command line argument like "comcast" or "giga"?

Yup. Case statement check for those two arguments.

> Oh man I gotta try this....
>
> In my ~/.slrnrc I can put my username and password for the server
> and it works when I use that server in my /etc/profile file. How do
> you handle the user/pass issue with your script?

I would have tried two lines of nnrpaccess, one for each server in
..slrnrc

> Do you just answer these questions when slrn loads?

Nope. No qwuestions needed.

> This is a bit difficult because giganews gave me a real
> weird username and password, I will never be able to remember these
> items.

Yes, I could not remember mine also.

> Can your script be modified to accept user/pass strings or can your
> script be modified to read them from a file?

Yes.

You may want to spend some time in
http://www.tldp.org/LDP/abs/html/index.html

I would use a nnrpaccess line for each server in ~/.slrnrc
Looking at man slrn you could have a .slrnrc_bell and .slrnrc_giga

and do something like

case $1 in
giga) NNTPSERVER=news.comcast.giganews.com
_rc_fn=$HOME/.slrnrc_giga
;;
bel) NNTPSERVER=news.bellsouth.whaterver
_rc_fn=$HOME/.slrnrc_giga
;;
*) echo " "
echo "$1 is an invalid selection"
echo " "
echo "Usage: news giga"
echo " news comcast"
exit 1
;;
esac

export NNTPSERVER

slrn -i $_rc_fn

>> xterm -tn xterm -fn 10x20 -title "news $1 " -geom 100x43+200+200 -e
>> slrn -C
>
> This last line I presume sets the window geometry when you use slrn in
> xwindows.

Almost right, xterm is an X terminal application running under my
desktop manager and I set the font and window position.

> What does "slrn -C" do?

Come on now, man slrn
look under OPTIONS

PS:
You said something about slrn.rc in one of your posts.
When slrn runs, it loads /etc/news/slrn.rc and then loads your
~/.slrnrc So if you duplicate lines in both you can trim your ~/.slrnrc
down to just your changes.

Once you get this to run you can create a aliases in .bashrc to call
news. Example:
alias slrnb='$HOME/scripts/news bell'
alias slrng='$HOME/scripts/news giga'

next login a slrnb would run slrn with bell's setup.

On Sun, 02 May 2004 20:20:50 GMT, Bit Twister wrote:

also you may need to add something like
server news.comcast.giganews.com .jnewsrc_giga
server netnews.comcast.net .jnewsrc_comcast
to ~/.slrnrc

Saw your other post about going back to windows for the solution.
Guess when longhorn comes out, you get to spend another 200 dollars. :)

In article <Xns94DDA5F5543DDmybigone@216.77.188.18>, Ohmster wrote:
> Bit Twister <BitTwister@localhost.localdomain> wrote in
> news:slrnc9af7e.n9h.BitTwister@wb.home.invalid:
>
>> As I read your original post, I thought slrnpull would be
>> consideration for off hour download.
>
> Doing it now with slrn. OMG, this is brutal. Might have to go back to Windows
> for this job...
>
> I really need a *good* binary news puller for this. Will have to search
> around a bit. Thanks Bit Twister.

I like nget. http://nget.sourceforge.net


--
http://home.mysth.be/~timvw

Bit Twister <BitTwister@localhost.localdomain> wrote in
news:slrnc9ambd.nqb.BitTwister@wb.home.invalid:

> Saw your other post about going back to windows for the solution.
> Guess when longhorn comes out, you get to spend another 200 dollars. :)

Heh heh, nah, not gonna happen. Having access to these large binary
newsgroups does have it's "advantages". Strictly for "research", of course.
;>)

Actually, I am quite thrilled with Windows XP Professional. Windows 3.1,
Win95, Win98, and Windows ME sucked because of all of the instability of the
OS. With previous versions of Windows, I had to reboot at least once a day,
sometimes more than that. XP runs forever and like *never* (Or hardly ever.)
crashes or becomes unstable, although some programs do crash from time to
time. I never turn off any of my computers, they run 24/7/52. I use auto
monitor off and sleeping hard drives to conserve power and prolong the lives
of the computers. XP and Linux do this quite well. I leave my Linux machine
logged out in run level 3 so that if I am not home and my girl's XP machine
takes a dump, she hops right on the redhat machine and logs in. She then
fires up xwindows in gnome and then runs her asmn program (Al's MSN Messenger
for Linux. This is quite a remarkable program, it is just like the real MSN
Messenger minus the audio/video chat.) She is a chat junkie and has a panic
attack if her machine goes down. Some nice chap in these newsgroups showed me
how to make the monitor blank and power down when it is inactive for 30
minutes in run level 3 and I placed those lines in my rc.local file. I also
put some stuff in there to boot with numlock on, both in xwindows and
console. This makes the monitor sleep when in run level 3:

# Make the monitor power down at console when not in use.
setterm -blank 5 -powersave off -powerdown 30

I also have a 21" monitor for my XP computer, it is nicer than the 17" one
that is on the Linux computer so I tend to actively use the XP computer more
than the linux one, althoug the Linux machine does it's job of being a
server, gateway, firewall all of the time. 21" CRT monitors are pretty cheap
when you know where to shop for them, you can get one for as little as $125,
I will get a larger monitor for the linux computer and maybe a long counter
top to put all three computers on to make it easier to use all of the
computers at once. Get my nice chair with wheels on it and a long runner to
go under the chairs so that I can scoot across to all of the computers. I ssh
into the Linux computer all the time, it is easier than going over to the
Linux machine and I use samba to map all of the important stuff over to the
XP machine. Using Dreamweaver this way is much better than trying to do the
html stuff directly on the Linux machine. There seems to be no really decent
Linux substitute for Dreamweaver. I also use xwin32 to log into the Linux
machine in gnome but using ssh is much faster and easier once you know the
console commands pretty well so I don't do that much anymore.

I will go back to my default bellsouth NNTP server for general newsgroup
reading with slrn. I had to dump slrn for the large binary stuff. I did
manage to pull the large binary group and man, slrn loaded all 2.5M headers
in record time, about a half hour or so. Then the threading and sorting, this
took quite a while, maybe 20 minutes, but the results were pretty
disappointing. I was running it from run level 3 and the headers in the
newsgroup are quite long, had to pan back and forth to read them. The binary
support in slrn is horrible, you get

fileA.rar (1 of 72) part 1/51
fileA.rar (1 of 72) part 2/51
fileA.rar (1 of 72) part 3/51...
fileB.rar (2 of 72) part 1/51
fileB.rar (2 of 72) part 2/51
fileB.rar (2 of 72) part 3/51...

etc., instead of

fileA.rar (1 of 72) 51/51 (all parts grouped in one line)
fileB.rar (2 of 72) 51/51 (all parts grouped in one line)
fileC.rar (3 if 72) 51/51 (all parts grouped in one line)

That was just plain insane. Can you imagine having to manually tag all of
these parts, in sequence, for each file set, and then again for all parts of
the set? And then what happens when the post in done in yenc? Udder madness I
tell you. Not worth the trouble. At least Pan and xnews group complete
binaries for you. slrn is a good text reader though. Pretty neat stuff.

I did find a program that looks quite promising for what I have in mind,
Binary News Reaper 2:

http://www.bnr2.org/

I downloaded the Linux version and installing it was a snap, it comes with a
linux installer. Just set the downloaded 14Mb file to executable and run it
with sh from a term window in xwindows. It will ask for a root passsword
during the install to put it's binary files where they must go. The program
is real sweet looking and just for binary newgroup harvesting. I am loading
the large binary newsgroup now, it is taking a real long time because it is
only downloading the headers as a single connection right at about 400Kbps
average. Not sure why slrn did this part so remarkably fast and other
programs take such a long time with this. This is not a problem because I
have full use of my XP machine for other stuff like playing "Far Cry" right
now. Game is rocking FPS with the most awesome graphics ever. BNR2 will make
8 connections for me when I get the newsgroup loaded and then let the reaping
begin!

I will give you the results of the Binary News Reaper experiment when I get
what I want from the newsgroup. BNR2 is at 90% now on the newsgroup headers,
will see how it goes when it is done downloading the headers, sorting, and
threading.

I am definitly going to use your cool as all hell script for slrn, thank you
so much for all of your terrrific help, Bit Twister. You "Da Man"!

--
~Ohmster

Tim Van Wassenhove <euki@pi.be> wrote in news:c73p1p$hp6j5$2@ID-
188825.news.uni-berlin.de:

> I like nget. http://nget.sourceforge.net

Will be checking that out, thanks Tim. Read my threads to Bit Twister to see
how Binary News Reaper 2 is coming along.

Thanks again, Tim.

--
~Ohmster

Tim Van Wassenhove <euki@pi.be> wrote in news:c73p1p$hp6j5$2@ID-
188825.news.uni-berlin.de:

> I like nget. http://nget.sourceforge.net

Wow, nget looks really good, Tim. Will definitely check that out. Thanks
buddy!

--
~Ohmster

On Sun, 02 May 2004 20:14:16 GMT, Ohmster
<bigbigkitty@invalid.anywhere.com> appears to have said:

> Bit Twister <BitTwister@localhost.localdomain> wrote in
> news:slrnc9af7e.n9h.BitTwister@wb.home.invalid:
>
>> As I read your original post, I thought slrnpull would be
>> consideration for off hour download.
>
> Doing it now with slrn. OMG, this is brutal. Might have to go back to Windows
> for this job...

If you aren't using the "jwk" patch, slrn will be a pain for big
multipart binaries. If you're comfortable patching and compiling
your own software, look here:

http://www.xs4all.nl/~thunder7/

In any event, with millions of headers, any screen or GUI based reader
I can think of is going to require *massive* amounts of memory and CPU
time for threading.

There's an excellent (IMO), low overhead solution, though... nget:

<http://nget.sourceforge.net/>

It's a bare bones command line app... it will "pseudo thread" the
multipart articles in a group, do XPAT searches, etc. I imagine it
will still take some horsepower to sort out those millions of headers
though. I would just use XPAT to search for what I want, and forget the
rest of the old stuff. It'll be reposted eventually, anyhow.

One caveat about nget. It's *not* intuitive. You must RTFM. And when
you are done, you must RTFM again. :)

HTH,
Jonathan

--
Don't just hit reply. Email address is broken. Thank
your friendly neighborhood spammer. Email replies to:
user: jnthn1 domain: earthlink<dot>net

"Jonathan A." <me@privacy.net> wrote in
news:slrnc9be4k.6kv.me@yomomma.example.net:

> There's an excellent (IMO), low overhead solution, though... nget:
>
> <http://nget.sourceforge.net/>
>
> It's a bare bones command line app... it will "pseudo thread" the
> multipart articles in a group, do XPAT searches, etc. I imagine it
> will still take some horsepower to sort out those millions of headers
> though. I would just use XPAT to search for what I want, and forget the
> rest of the old stuff. It'll be reposted eventually, anyhow.
>
> One caveat about nget. It's *not* intuitive. You must RTFM. And when
> you are done, you must RTFM again. :)
>
> HTH,
> Jonathan

Yeah that sounds really good. I was just at the web site and this little gem
looks like a real keeper. I installed Binary News Reaper 2 and man, that
really worked swell. Got the job done and I will be keeping that application.

Definitely want to try out nget though, sounds like something I really would
need. Thanks for the advice, Johnathan.

--
~Ohmster

"Jonathan A." <me@privacy.net> wrote in
news:slrnc9be4k.6kv.me@yomomma.example.net:

> If you aren't using the "jwk" patch, slrn will be a pain for big
> multipart binaries. If you're comfortable patching and compiling
> your own software, look here:
>
> http://www.xs4all.nl/~thunder7/
>
> In any event, with millions of headers, any screen or GUI based reader
> I can think of is going to require *massive* amounts of memory and CPU
> time for threading.

Okay, I'll bite. It is late now and I am going to bed. There is an excellent
new special on The Discovery Channel, "Dive To The Bermuda Triangle" so I am
going to check out now.

I have:
[ohmster@ohmster ohmster]$ rpm -q slrn
slrn-0.9.7.4-9

....installed so it might be time to pull up the sleeves, rpm -e slrn away,
download slrn-0.9.7.4.tar.gz (If I can find it.), apply the patch, and
compile. I probably won't use this much as Binary News Reaper 2 for xwindows
did such a good job on this totally massive newsgroup, and nget really seems
like the way to go for the low overhead console, but this would be a totally
excellent experiment, non the less, I would think.

BNR2 did not choke on the 2.5M headers. It launched 8 connections, 7 of them
died idle during the header pull and this took a long time, over 2 hours.
Once the headers came down and I refreshed the window, everything was right
there. As Emeril would say, "Bam!". Used the easy filter to find what I
wanted, queued them up and downloaded. The 8 connections roared to life and
the download of 150Mb was done in like no time. Amazing. xnews and Windows XP
choked to death on this on a 2Ghz P4 1Gb DDRRAM machine. BNR2 and linux had
no problems at all on a PIII 800Mhz 1.5Gb SDRAM machine. Go figure.

Thanks for the very helpful advice, Johnathan.

--
~Ohmster