OK to have many files in one folder?

Server is running Linux, and PHP is constantly creating and modifying
images in a directory. Apache is constantly serving these same
images. There are about 250,000 of them in the same directory. Seems
to be running OK, no problematic CPU load, but I'm wondering if
anyone knows whether I'm living dangerously having that many docs in
one directory with that much activity. It is an extremely busy server.

Sorry is this seems like more of a linux sysad question than a PHP
question. Thanks... :)

[snip]
Server is running Linux, and PHP is constantly creating and modifying
images in a directory. Apache is constantly serving these same
images. There are about 250,000 of them in the same directory. Seems
to be running OK, no problematic CPU load, but I'm wondering if
anyone knows whether I'm living dangerously having that many docs in
one directory with that much activity. It is an extremely busy server.

Sorry is this seems like more of a linux sysad question than a PHP
question. Thanks... :)
[/snip]

Two words. Beware the inode.

6/15/07, Jay Blanchard <jblanchard@pocket.com> wrote:
> [snip]
> Server is running Linux, and PHP is constantly creating and modifying
> images in a directory. Apache is constantly serving these same
> images. There are about 250,000 of them in the same directory. Seems
> to be running OK, no problematic CPU load, but I'm wondering if
> anyone knows whether I'm living dangerously having that many docs in
> one directory with that much activity. It is an extremely busy server.
>
> Sorry is this seems like more of a linux sysad question than a PHP
> question. Thanks... :)
> [/snip]
>
> Two words. Beware the inode.
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

If that directory is all on one single partition, God help you if
there's any corruption. Check into RAID 0+1/RAID 5 disk striping if
you've got that many images. And remember, the fact that they're all
in one directory doesn't matter at all to the system, as directories,
folders, et cetera, are just representations for human readability and
organization. In fact, those files reside on several sectors
throughout the drive, and each file itself is probably fragmented many
times.

Just for fun, though, if you want to crash your server, you could always do:
`while [ 1 = 1 ]; do ls -l;done`

--
Daniel P. Brown
[office] (570-) 587-7080 Ext. 272
[mobile] (570-) 766-8107

On Saturday 16 June 2007 02:51, Daniel Brown wrote:

> And remember, the fact that they're all
> in one directory doesn't matter at all to the system, as directories,
> folders, et cetera, are just representations for human readability and
> organization. In fact, those files reside on several sectors
> throughout the drive, and each file itself is probably fragmented many
> times.

Actually it does matter depending on the filesystem you use. If you're
using ext2/ext3 then having several thousand files in a directory
seriously slows things down.

--
Crayon

Thanks for the replies but I'm not sure I know what to do with them.
Is the problem with the number of files, or is the problem with the
activity? Can you dumb down your answers at all for me? :)


On Jun 15, 2007, at 11:51 AM, Daniel Brown wrote:

> 6/15/07, Jay Blanchard <jblanchard@pocket.com> wrote:
>> [snip]
>> Server is running Linux, and PHP is constantly creating and modifying
>> images in a directory. Apache is constantly serving these same
>> images. There are about 250,000 of them in the same directory. Seems
>> to be running OK, no problematic CPU load, but I'm wondering if
>> anyone knows whether I'm living dangerously having that many docs in
>> one directory with that much activity. It is an extremely busy
>> server.
>>
>> Sorry is this seems like more of a linux sysad question than a PHP
>> question. Thanks... :)
>> [/snip]
>>
>> Two words. Beware the inode.
>>
>> --
>> PHP General Mailing List (http://www.php.net/)
>> To unsubscribe, visit: http://www.php.net/unsub.php
>>
>>
>
> If that directory is all on one single partition, God help you if
> there's any corruption. Check into RAID 0+1/RAID 5 disk striping if
> you've got that many images. And remember, the fact that they're all
> in one directory doesn't matter at all to the system, as directories,
> folders, et cetera, are just representations for human readability and
> organization. In fact, those files reside on several sectors
> throughout the drive, and each file itself is probably fragmented many
> times.
>
> Just for fun, though, if you want to crash your server, you
> could always do:
> `while [ 1 = 1 ]; do ls -l;done`
>
> --
> Daniel P. Brown
> [office] (570-) 587-7080 Ext. 272
> [mobile] (570-) 766-8107
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>

On 6/15/07, Crayon Shin Chan <crayon.shin.chan.uk@gmail.com> wrote:
> On Saturday 16 June 2007 02:51, Daniel Brown wrote:
>
> > And remember, the fact that they're all
> > in one directory doesn't matter at all to the system, as directories,
> > folders, et cetera, are just representations for human readability and
> > organization. In fact, those files reside on several sectors
> > throughout the drive, and each file itself is probably fragmented many
> > times.
>
> Actually it does matter depending on the filesystem you use. If you're
> using ext2/ext3 then having several thousand files in a directory
> seriously slows things down.
>
> --
> Crayon
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

Once again, this doesn't matter so much for per-directory (though
listing will take longer, as I think I mentioned) as it does the
filesystem mount. The ext2/ext3 filesystems were made for these
reasons, especially with journaling like ReiserFS, XFS, et cetera
(which is a completely different bag of nuts).

--
Daniel P. Brown
[office] (570-) 587-7080 Ext. 272
[mobile] (570-) 766-8107

I can easily break it up into 100 subdirectories, 2500 files in each,
would that be good insurance against problems?

[snip]
I can easily break it up into 100 subdirectories, 2500 files in each,
would that be good insurance against problems?
[/snip]

As someone mentioned, directories are just a human convenience. Each
file will have an inode and is identified by an inode number in the file
system where it resides. Inodes store information on files such as user
and group ownership, access mode (read, write, execute permissions) and
type of file. There is a fixed number of inodes, which indicates the
maximum number of files each filesystem can hold. Hence, beware the
inode.

On 6/15/07, Brian Dunning <brian@briandunning.com> wrote:
> I can easily break it up into 100 subdirectories, 2500 files in each,
> would that be good insurance against problems?
>

I have no clue how big the files are, but you might want to store them
in a database. That can speed up things, but don't ask me how much ;)

Tijnema

On 6/15/07, Brian Dunning <brian@briandunning.com> wrote:
> Thanks for the replies but I'm not sure I know what to do with them.
> Is the problem with the number of files, or is the problem with the
> activity? Can you dumb down your answers at all for me? :)
>
>
> On Jun 15, 2007, at 11:51 AM, Daniel Brown wrote:
>
> > 6/15/07, Jay Blanchard <jblanchard@pocket.com> wrote:
> >> [snip]
> >> Server is running Linux, and PHP is constantly creating and modifying
> >> images in a directory. Apache is constantly serving these same
> >> images. There are about 250,000 of them in the same directory. Seems
> >> to be running OK, no problematic CPU load, but I'm wondering if
> >> anyone knows whether I'm living dangerously having that many docs in
> >> one directory with that much activity. It is an extremely busy
> >> server.
> >>
> >> Sorry is this seems like more of a linux sysad question than a PHP
> >> question. Thanks... :)
> >> [/snip]
> >>
> >> Two words. Beware the inode.
> >>
> >> --
> >> PHP General Mailing List (http://www.php.net/)
> >> To unsubscribe, visit: http://www.php.net/unsub.php
> >>
> >>
> >
> > If that directory is all on one single partition, God help you if
> > there's any corruption. Check into RAID 0+1/RAID 5 disk striping if
> > you've got that many images. And remember, the fact that they're all
> > in one directory doesn't matter at all to the system, as directories,
> > folders, et cetera, are just representations for human readability and
> > organization. In fact, those files reside on several sectors
> > throughout the drive, and each file itself is probably fragmented many
> > times.
> >
> > Just for fun, though, if you want to crash your server, you
> > could always do:
> > `while [ 1 = 1 ]; do ls -l;done`
> >
> > --
> > Daniel P. Brown
> > [office] (570-) 587-7080 Ext. 272
> > [mobile] (570-) 766-8107
> >
> > --
> > PHP General Mailing List (http://www.php.net/)
> > To unsubscribe, visit: http://www.php.net/unsub.php
> >
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

Brian,

I stand behind my suggestion, though I also suggest you take
whatever everyone else says into consideration just as much as me. I
have well over a decade of experience, but I'll be the first to admit
that sometimes a teenage computer geek is more correct than I am.

Yes, list, believe it or not, I'm not always correct. I'm just
going to let that sink in a little bit.

Okay, now that everyone is done laughing at me (well, mostly
everyone), let's get on with it.

I still think it's best to split the files across a RAID. Ask
your datacenter to discuss the pros and cons of both RAID 0+1 and RAID
5 with you, and research it on the web. It would be better to split
the files into different directories, as well, but *probably* is not
necessary. Take that as you will.

That said, one way of doing it that is very common is to create a
minimum of 36 directories, from 0 - 9 and a - z. Each directory
should contain only files where the first character is the same as the
directory within which it resides. However, if you have a specific
character that comes up much more frequently than others, or even all
the time, this won't work, of course.

The point is, in my experience, holding even a terabyte of data in
smaller files within one directory is not a Bad Thing [tm], per se.
I've administered MySQL databases on servers that were in the
two-to-three terabyte range, and one of which had well over 3,000
separate tables. Each of those tables is three separate files
(minimum) on disk, mind you: table.frm, table.MYD, table.MYI. Also,
keep in mind that I didn't create the database structure, but was
instead hired to ensure that it ran smoothly. In the four months and
n-million queries that were run on it while it was in use, the server
responded as if it were a standard 50MB database in under a dozen
files (if memory serves, it was a Dual Xeon 2.6GHz with 4GB RAM) and
only used for MySQL). After the project was completed, however, we
did upgrade the systems to a distributed network for further
expansion, but I think you get the point there.

Bottom line, however, is that just because you're not experiencing
problems now doesn't mean you won't soon in the future. Seriously
consider, with that many separate files and that much data, how much
it's worth to you in the long run, and if a RAID setup and
directory-splitting is a Good Thing [tm] for your situation.

--
Daniel P. Brown
[office] (570-) 587-7080 Ext. 272
[mobile] (570-) 766-8107