OPENDIR number of files problem

[Bill Campbell]

On Wed, Oct 08, 2008, John Esak wrote:
>Wow! And I'm always complaining about our filepro folders with 2,800 files
>in them!  But 7,000!  Wow!
>
>For everyone who doesn't know.  The file systems used on Unix and Windows
>are not good (in retrieval speed, etc.) with folders that have more than 999
>files in them....

While I agree that directories with large numbers of entries are
a Bad Idea(tm), access speed can be fine depending on the type of
file system.  I don't claim to be an expert on the guts of
various file systems, but several non-SCO *nix file systems
maintain btree indexes to directory entries which can be quite
fast, and do not exhibit the quadratic (or whatever) slow downs
seen in early *nix file systems.

SCO OpenServer had a hard limit of 999 directories in the root
directory of any file system.  I first ran into this at an ISP
site when they attempted to add their 1,000th customer to the
/home directory.  This limit does (did) not apply to directories
below the root of a file system.

FWIW, My main e-mail and file server here has /home mounted on an
xfs file system, and my Maildir folder for security messages has
had more than 40,000 messages this month (it automatically
archives any over 30 days old, and is ``down'' to 36,306 right
now).  It takes about two seconds for mutt to build the index
display of these messages when run directly on the file system
(you don't want to try opening that folder with Thunderbird or
other IMAP mail clients :-).

Just for fun, I ran a simple python script that uses the
os.walk() function to recursively parse everything in this
Maildir directory and get the total file sizes. with the
following results.  This routine basically gets the list of
directory entries, the does a stat() on each entry to get the
size.  It does not actually open any of the files.