Efficiency of file vs database

Ok, I have a program that reads the contents of a file (1 line, 5 '|'
seperated items). Every so often (between twice a day, and 200 times a
day), it rewrites the contents of that file. I also do a few database
update queries whenever the file is written. I only open up the
database connector if I need to update the database. The script itself
runs every 30 seconds. My question, is that is it faster and more
efficient to read the file every time, or open the database connector
and make a query? The file name never changes, and I read the whole
file at one time (679 bites).

ircmaxell wrote:

> Ok, I have a program that reads the contents of a file (1 line, 5 '|'
> seperated items). Every so often (between twice a day, and 200 times a
> day), it rewrites the contents of that file. I also do a few database
> update queries whenever the file is written. I only open up the
> database connector if I need to update the database. The script itself
> runs every 30 seconds. My question, is that is it faster and more
> efficient to read the file every time, or open the database connector
> and make a query? The file name never changes, and I read the whole
> file at one time (679 bites).

Hi,

Only one way to find out: timestamp both operations and see how long they
take to complete.
But the results may vary a lot based on machineload and needed diskIO (for
other processes).

In general: If your application is running fine with a small file, leave it
that way.
A database will typically be faster if your file is big, because opening a
big file and loading it into memory will take diskIO time, which is often a
bottleneck on modern machines (fast CPU, relatively slow disk IO).

More general: If you don't have a problem, don't fix it. :-)
You should of course be carefull when writing to a file and take precautions
no other process is doing the same (filelocking).

just my 2 cent.

Regards,
Erwin Moller

Following on from ircmaxell's message. . .
>Ok, I have a program that reads the contents of a file (1 line, 5 '|'
>seperated items). Every so often (between twice a day, and 200 times a
>day), it rewrites the contents of that file. I also do a few database
>update queries whenever the file is written. I only open up the
>database connector if I need to update the database. The script itself
>runs every 30 seconds. My question, is that is it faster and more
>efficient to read the file every time, or open the database connector
>and make a query? The file name never changes, and I read the whole
>file at one time (679 bites).
>

Well since you have both methods at your disposal why don't you compare
them? If you can't detect a speed difference then it doesn't matter.

In my experience this sort of choice will probably depend on other
things anyway.

I have a feeling that you're making a mountain out of a molehill of
machine cycles.

--
PETER FOX Not the same since the porcelain business went down the pan
peterfox@eminent.demon.co.uk.not.this.bit.no.html
2 Tees Close, Witham, Essex.
Gravity beer in Essex <http://www.eminent.demon.co.uk>

Well, the problem I have, is that durring the peak times, the server is
at its limit of resouces (CPU and Ram) and getting around 100 requests
per second (images included). So if I could save a tiny bit, it would
make a difference... Even at its peak, the script runs every 30
seconds, however if I could reduce the memory footprint created by this
program, it would reduce Apache's child process memory footprints...
Just a question... Thanks!

On Dec 18, 6:20 am, Peter Fox
<peter...@eminent.demon.co.uk.not.this.bit.no.html > wrote:
> Following on from ircmaxell's message. . .
>
> >Ok, I have a program that reads the contents of a file (1 line, 5 '|'
> >seperated items). Every so often (between twice a day, and 200 times a
> >day), it rewrites the contents of that file. I also do a few database
> >update queries whenever the file is written. I only open up the
> >database connector if I need to update the database. The script itself
> >runs every 30 seconds. My question, is that is it faster and more
> >efficient to read the file every time, or open the database connector
> >and make a query? The file name never changes, and I read the whole
> >file at one time (679 bites).Well since you have both methods at your disposal why don't you compare
> them? If you can't detect a speed difference then it doesn't matter.
>
> In my experience this sort of choice will probably depend on other
> things anyway.
>
> I have a feeling that you're making a mountain out of a molehill of
> machine cycles.
>
> --
> PETER FOX Not the same since the porcelain business went down the pan
> peter...@eminent.demon.co.uk.not.this.bit.no.html
> 2 Tees Close, Witham, Essex.
> Gravity beer in Essex <http://www.eminent.demon.co.uk>

Erwin Moller wrote:
> In general: If your application is running fine with a small file, leave it
> that way.
> A database will typically be faster if your file is big, because opening a
> big file and loading it into memory will take diskIO time, which is often a
> bottleneck on modern machines (fast CPU, relatively slow disk IO).
>

Building upon that... The file may be small enough right now for it
not to matter. But if it starts to get bigger in the future and you've
already gone heavily down this path, you may need to do significant
backtracking to switch over to a database.

Just something to keep in mind.

ircmaxell wrote:

> Well, the problem I have, is that durring the peak times, the server is
> at its limit of resouces (CPU and Ram) and getting around 100 requests
> per second (images included). So if I could save a tiny bit, it would
> make a difference... Even at its peak, the script runs every 30
> seconds

If the script runs every 30 seconds, but you get 100 requests per second,
that means that only 0.3% of requests are for this particular script.
Which means that you might be better off optimising the other 99.7% of the
requests!

Although Apache is a lovely server, there are some others which might run
faster for you if you don't need some of the extra features that Apache
provides. Take a look at, say, "thttpd" -- your PHP will likely run slower
(as it runs PHP through CGI instead of mod_php) but the speed-up you get
for other files may more than make up for it.

--
Toby A Inkster BSc (Hons) ARCS
Contact Me ~ http://tobyinkster.co.uk/contact

Well, let me clarify... The site is comprised of a complete php
frontend (Joomla). There are 4 HTML files and an XML file that recieve
90% of the traffic (the other 10% accounts for enough traffic to keep
php running fast). The script that we are talking about rewrites those
5 files (plus a tiny one) on every run. I need to use apache for
mod_rewrite and other functions on other domains hosted.
The site : www.bagsofcrap.com
One HTML file http://www.bagsofcrap.com/woot.html
XML file http://www.bagsofcrap.com/woot.xml
Small txt file http://www.bagsofcrap.com/wootdata.txt

I would post the update script, but that's pointless...

On a side note. Would it be worth it to not write 4 HTML files, but
have them hit a small php file instead that queries a database for the
data? Or is my way most efficient?
Toby Inkster wrote:
> ircmaxell wrote:
>
> > Well, the problem I have, is that durring the peak times, the server is
> > at its limit of resouces (CPU and Ram) and getting around 100 requests
> > per second (images included). So if I could save a tiny bit, it would
> > make a difference... Even at its peak, the script runs every 30
> > seconds
>
> If the script runs every 30 seconds, but you get 100 requests per second,
> that means that only 0.3% of requests are for this particular script.
> Which means that you might be better off optimising the other 99.7% of the
> requests!
>
> Although Apache is a lovely server, there are some others which might run
> faster for you if you don't need some of the extra features that Apache
> provides. Take a look at, say, "thttpd" -- your PHP will likely run slower
> (as it runs PHP through CGI instead of mod_php) but the speed-up you get
> for other files may more than make up for it.
>
> --
> Toby A Inkster BSc (Hons) ARCS
> Contact Me ~ http://tobyinkster.co.uk/contact

Usually when in need to compare which is faster, I tend to think that
in the very end databases are just flat files. The only difference
between real flat files and database is the database software, which
has been built to optimize, among other things, the speed of quering of
the data.

This of course might not be the case in your problem, since reading
just a simple file might be faster. However if the database software is
in the same server with the php, the connection should be very fast.

Just babbling, try it out. As someone mentioned, if your file grows
you'll might be at risk in the future. Databases are just so much more
scalable.

jussi

On Tue, 19 Dec 2006 11:59:29 -0800, jussist@gmail.com wrote:

> Usually when in need to compare which is faster, I tend to think that in
> the very end databases are just flat files. The only difference between
> real flat files and database is the database software, which has been
> built to optimize, among other things, the speed of quering of the data.

If you disregard all of the fundamentals of database design...
normalization, index optimization, etc.

A flat file can be very fast as long as you're talking about a very small
file. At the point the overhead of the database server matches the access
time of the flat file based on its size the database server will always be
faster... assuming your database isn't designed like a flat file (which it
shouldn't be).

Well, I wound up redesigning the program anyway, and what a speed
difference!!! I went to an almost completely database system (I still
write a bunch of files, but I don't read any of them). The page load
went from .93 seconds to .24 seconds... The file read portion went
from .36 seconds to (are you ready?) a measly .00023 seconds using the
database... WOW!!! You can say that is a little difference...

On Dec 19 2006, 3:29 pm, Ivan Marsh <anno...@you.now> wrote:
> On Tue, 19 Dec 2006 11:59:29 -0800, juss...@gmail.com wrote:
> > Usually when in need to compare which is faster, I tend to think that in
> > the very end databases are just flat files. The only difference between
> > real flat files and database is the database software, which has been
> > built to optimize, among other things, the speed of quering of the data.If you disregard all of the fundamentals of database design...
> normalization, index optimization, etc.
>
> A flat file can be very fast as long as you're talking about a very small
> file. At the point the overhead of the database server matches the access
> time of the flat file based on its size the database server will always be
> faster... assuming your database isn't designed like a flat file (which it
> shouldn't be).