About chdir... and directory/file storage

[Bill Vermillion]

On Thu, Sep 02 06:40  John Esak said 'Who you talkin' to? You talkin'
to John Esak?  I didn't do nuttin'.  I said: 


> Richard (and others),

....

> The problem of EDI and storing so much incoming data is very
> important. It is necessary to store the files that arrive daily
> in some manner that makes sense and can be dealt with easily as
> time goes by, days, months, years. That is why I took some time
> to develop this scheme. It has saved me untold headaches over
> the past several years. I have to deal with both DOS and Unix
> so it deals with the slash/backslash thing in a way that makes
> sense to me, in that it works on both sides. Admittedly, there
> are many different ways to handle this and there are times when
> a / will work equally well on both platforms... still, this is
> how I handled it. More to the point of the actual traversing of
> directories, I adopted the following procedure.

> On any day that files come into the system, I put them in a directory with a
> hierarchy of that year, that month and then that day with a batch number
> from 1 to 100, so that multiple incoming files for each day can be
> accommodated. To do this, I have filePro check to see if the full dirpath is
> already there, if so, I use it. If not, I create the path, whether it be
> just to add a new month to the current year, or to add a new year if it
> doesn't exist and then add the new month as well. Happily, the system has
> been working for quite some time now without any attendance from me. I have
> directories (and files) which look like this now and it is very easy to
> traverse them with filePro processes _and_ human-applied searches with "cd".
> I can find anything quickly by date and that is the name of the game. (Also,
> more importantly, they can be traversed by either Windows filePro or Unix
> filePro.) These are not exact portrayals of the directories created in the
> processing, but close, just to give you the idea...

>     /u/edi/ups/arch/inv/01/01/invbat01.dat
>     /u/edi/ups/arch/inv/01/01/invbat02.dat
>     /u/edi/ups/arch/inv/01/02/invbat01.dat
>     /u/edi/ups/arch/inv/01/02/invbat01.dat
>     /u/edi/ups/arch/inv/01/12/invbat01.dat
>     /u/edi/ups/arch/inv/02/01/invbat01.dat
>     /u/edi/ups/arch/inv/02/01/invbat02.dat
>     /u/edi/ups/arch/inv/02/02/invbat01.dat
>     /u/edi/ups/arch/inv/02/02/invbat01.dat
>     /u/edi/ups/arch/inv/02/12/invbat01.dat
> and
>     /u/edi/ups/arch/po/01/01/invbat01.dat
>     /u/edi/ups/arch/po/01/01/invbat02.dat
>     /u/edi/ups/arch/po/01/02/invbat01.dat
>     /u/edi/ups/arch/po/01/02/invbat01.dat
>     /u/edi/ups/arch/po/01/12/invbat01.dat
>     /u/edi/ups/arch/po/02/01/invbat01.dat
>     /u/edi/ups/arch/po/02/01/invbat02.dat
>     /u/edi/ups/arch/po/02/02/invbat01.dat
>     /u/edi/ups/arch/po/02/02/invbat01.dat
>     /u/edi/ups/arch/po/02/12/invbat01.dat
> 

Let me comment on what John has said is almost a must.  I can't
emphasize that too strongly.

I had one client whose SW author would store a great many daily
files [not EDI but faxes] in one directory.

When you have a dual 800MHz Xeon system, with 1GB RAM, and 
a high-speed RAID with 15K Cheetah drives, and a directory
listing takes about three minutes you can see what a bad design
deicsion that was.   That system was very fast compared to the old
system.  Their Friday end-of-week report used to take about 2
hours, and the new machine ran it in 7 minutes.  So when they
decided to store all the fax data the non-Unix SW designer made a
decision that caused parts of the system to perform as poorly
as a 100MHz Pentium with only 64K of RAM. 

Once you get more than 500 files in a directory you start taking
performance hits.  On a very high performance machine with very
fast drives you can get more files before you start hearing about
performance hits, but good design means you'll never have those
complaints based on your work.

Aside:  How many remember DOS 1 that had a limit of just
over 500 files TOTAL.  That's why DOS 2 had the hierarchical model
based loosely on Unix to permit more files.

Bill
-- 
Bill Vermillion - bv @ wjv . com