OPENDIR number of files problem

[John Esak]

Ah, thanks Bill.  It's good to know some of this stuff.  We are wofking
mostly on Linux with these large clients.... well, take that back... several
are on Linux, the majority are on 2003 server.... yes, I know how you just
cringed and made the sign of the crosss. :-)  But, it's life in this
state/county's mandates.  I've been learning a lot about the server stuff,
but from what I see, the ntfs file system is not great with large nubmers of
files in a directory.  But, the machine I was having some slowness (huge
slowness) with today was a Buffalo TeraStation that employs some (red hat
looking) flavor of Linux.

Anyway, it's nice to know that on some Linux systems, at least, the <999
convention is not critical.

John


> -----Original Message-----
> From: filepro-list-bounces+john=valar.com at lists.celestial.com
> [mailto:filepro-list-bounces+john=valar.com at lists.celestial.com] On Behalf
> Of Bill Campbell
> Sent: Wednesday, October 08, 2008 7:55 PM
> To: filepro-list at lists.celestial.com
> Subject: Re: OPENDIR number of files problem
> 
> On Wed, Oct 08, 2008, John Esak wrote:
> >Wow! And I'm always complaining about our filepro folders with 2,800
> files
> >in them!  But 7,000!  Wow!
> >
> >For everyone who doesn't know.  The file systems used on Unix and Windows
> >are not good (in retrieval speed, etc.) with folders that have more than
> 999
> >files in them....
> 
> While I agree that directories with large numbers of entries are
> a Bad Idea(tm), access speed can be fine depending on the type of
> file system.  I don't claim to be an expert on the guts of
> various file systems, but several non-SCO *nix file systems
> maintain btree indexes to directory entries which can be quite
> fast, and do not exhibit the quadratic (or whatever) slow downs
> seen in early *nix file systems.
> 
> SCO OpenServer had a hard limit of 999 directories in the root
> directory of any file system.  I first ran into this at an ISP
> site when they attempted to add their 1,000th customer to the
> /home directory.  This limit does (did) not apply to directories
> below the root of a file system.
> 
> FWIW, My main e-mail and file server here has /home mounted on an
> xfs file system, and my Maildir folder for security messages has
> had more than 40,000 messages this month (it automatically
> archives any over 30 days old, and is ``down'' to 36,306 right
> now).  It takes about two seconds for mutt to build the index
> display of these messages when run directly on the file system
> (you don't want to try opening that folder with Thunderbird or
> other IMAP mail clients :-).
> 
> Just for fun, I ran a simple python script that uses the
> os.walk() function to recursively parse everything in this
> Maildir directory and get the total file sizes. with the
> following results.  This routine basically gets the list of
> directory entries, the does a stat() on each entry to get the
> size.  It does not actually open any of the files.
> 
> _______________________________________________
> Filepro-list mailing list
> Filepro-list at lists.celestial.com
> http://mailman.celestial.com/mailman/listinfo/filepro-list