Guru Magazine GL Article

John Esak john at valar.com
Sun Jul 27 10:36:23 PDT 2008 

Thanks Soott,
Yeah, too bad there isn't something like it around today... I would actually
love to see a Linux mag just like it.  And/or I have always wanted the FPDJ
back.

John

> -----Original Message-----
> From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> Sent: Sunday, July 27, 2008 1:22 PM
> To: john at valar.com
> Cc: filepro-list at lists.celestial.com
> Subject: Guru Magazine GL Article
> 
> John,
> 
> I am in the process of writing a GL Module for my application.  I just
> wanted to let you know how helpful your Guru magazine article of March
> 1991 was.  Great overview of what you need to think about before
> starting the coding.  It's amazes me how useful that old Guru rag is
> after all these years.
> 
> Regards,
> 
> Scott
> 
> 
> Scott Walker
> RAM Systems Corp.
> ScottWalker at RAMSystemsCorp.com
> Ph: (704) 896-6549
> 
> 
> 
> 
> -----Original Message-----
> From: John Esak [mailto:john at valar.com]
> Sent: Sunday, July 27, 2008 1:06 PM
> To: 'Scott Walker'
> Cc: filepro-list at lists.celestial.com
> Subject: RE: Free Disk Necessary To Retructure A File
> 
> 
> Oh my god... let me find a recent email about that... and I'll send it
> up.
> 
> The short answer is we had a 3Gb CPU, 300Gb Sata 3.0, 2Gb RAM system
> working on a call center app.  14 people using a FacetPhone system to
> receive calls and store the data on the machine and each appropriate
> scheduled time, etc. (the FacePhone being used is running on another
> Linux server.. .unimportant to this setup.)
> 
> Anyway, the system was SO slow that almost every operation was
> noticeably slowed down... usually filePro operations, reports, etc., are
> pretty instantaneous. On a rig like this.  However, even with no one
> else running, it would take 27 minutes to run all the indexes on the 15
> files involved.
> 
> We put together a 2.4Gb CPU, 1Tb RAID 5, 2Gb RAM running Linux ubuntu.
> The system makes use of a "software" based RAID provided by the Linux.
> (Yech,,, I thought)  But, performance change was beyond staggering.  For
> example, and I swear this is true... the complete index build now takes
> 12.12 seconds!!!!!  That is just staggering, and every other operation
> dealing with file I/O is that much improved as well.  So, if you can put
> SATA in a RAID, preferably a true hardware RAID, not the "fake" RAID
> some motherboards supply, you will get great performance for a small
> group of users, like 10 to 20.  Moor than that I can't verify.
> 
> As for this SATA software RAID vs a true battery-backed up hardware SCSI
> RAID a la what you would get in a new HP server, or even a DELL... the
> performance would actually be even better (and more reliable) than this.
> Plus, the performance would NOT degrade as you add users, up to probably
> a couple hundred.  By the way, both machines in the above example are
> Core II Duos.  And as an aside, when we watch the CPU usage on the SCO
> side the first single drive box, it was not too exciting. Only a little
> bit of sporadic uneven usage of both CPU's was noticed.  On the Linux
> box, the whole operation, everything, system calls, file I/O, everything
> causes both CPU's to run almost all the time in a very even and smooth
> manner.  The Linux obviously runs the threaded and both threaded
> programs (like filePro) in a much more sophisticated way.  The downside
> is that with this software RAID from the ubuntu driver, the CPU's are
> used elaborately during all file I/O to the RAID.  This would be almost
> zero usage if we were using a good battery backed up hardware adaptor
> for a SCSI setup.
> 
> So, to sum up, I have radically mixed apples and oranges by comparing
> RAID to a single drive and not SCSI single drive to single drive.
> However, I wanted to make you aware that you *can* make use of these
> inexpensive SATA drives for a filePro system with a limited number of
> users... probably up to 50.  there are many tests showing SCSI vs SATA
> in single to single comparisons.. and they are pretty even, with SATA
> even being faster in some instances.  This is why I tried putting our
> application on the single SATA drive in the first place.  It was only
> when we actually experienced the TERRIBLE slow down when all 14 users
> were running rclerk on some high activity file I/O that it became sadly
> very clear to me that SATA is nothing like SCSI.  A single SCSI drive
> can easily handle any filePro app with only 14 users.  I have
> successfully put 40 to 70 users on a single SCSI drive running filePro
> apps.... it's not my choice, but it works.  This pore little SATA drive
> couldn't have handled even 15 users, 1 more above the 14... I fear that
> would have broken the camel's back.  Indeed, we had some pretty good
> size glitches even as it was.  The RAIDing of 5 drives in RAID 5 fixed
> this all... and we didn't need to hit the big expense of SCSI drives and
> an adaptor.  But, remember, that we will not try and put more than 15
> people or so on even this system.  Before too long we'll build a machine
> that has a true SCSI or SASI RAID.  We actually halved several rack
> mounted systems like this already, but they are designated for other
> customers and applications, too bad.
> 
> Hope I helped you understand the somewhat subtle differences between
> SATA and SCSI.  Main difference if I didn't hit is that the SCSI drives
> each have their own sophisticated controllers on board, and the
> adaptor/controller itself is also super sophisticated.... all of this
> means the CPU's have to do very little work compared to their constant
> bit by bit work for a SATA drive.  This is an easily noticeable
> difference believe me.
> 
> No time to spell check all these words... .sorry....
> 
> John
> 
> > -----Original Message-----
> > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > Sent: Saturday, July 26, 2008 8:58 PM
> > To: john at valar.com
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> > Is the performance of SCSI vastly superior over SATA in a Linux server
> 
> > with multiple users?
> >
> > Scott Walker
> > RAM Systems Corp.
> > ScottWalker at RAMSystemsCorp.com
> > Ph: (704) 896-6549
> >
> >
> >
> >
> > -----Original Message-----
> > From: John Esak [mailto:john at valar.com]
> > Sent: Saturday, July 26, 2008 7:55 PM
> > To: 'Scott Walker'
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> >
> > Yeah, this is little known ... so I'm glad it helps you out.  However,
> 
> > customers should *always* upgrade their servers!  :-)  And, seriously,
> 
> > if they are using a system with only 400MB free these days... that is
> > not enough... just on GP's.  Everyone talks about needing 10% free
> > disk space at the very least.  I like to have 50% free whenever I can
> > get it.. .and today with 500Gb SATA drives costing $105!!!! Well, it's
> 
> > cheaper and easier to have lots of space than it ever was.  By the
> > way, I still and will probably forever support and suggest SCSI over
> > SATA, just using that as an example.
> >
> >
> > John
> >
> >
> > > -----Original Message-----
> > > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > > Sent: Saturday, July 26, 2008 7:48 PM
> > > To: john at valar.com
> > > Cc: filepro-list at lists.celestial.com
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > > John,
> > >
> > > Thank you much.  I never realized that's how it worked.  I always
> > > thought if you had a 500mb file and you were adding 100mb to it,
> > > then you would need 1.1gb free to do it.  I thought it copied stuff
> > > from the "old" key file to a new bigger one, and then deleted the
> > > old key file. Thus since there would be an instant in time where
> > > both the "old" key file existed and the new one existed, just before
> 
> > > it deleted
> >
> > > the old one I would need enough disk space for both.
> > >
> > > This is really a case of the customer causing the problem.  I have
> > > been telling them for (3) years that they must upgrade their server.
> > >
> > > Thanks again for the info.
> > >
> > > Regards,
> > >
> > > Scott
> > >
> > > Scott Walker
> > > RAM Systems Corp.
> > > ScottWalker at RAMSystemsCorp.com
> > > Ph: (704) 896-6549
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Esak [mailto:john at valar.com]
> > > Sent: Saturday, July 26, 2008 7:31 PM
> > > To: 'Scott Walker'
> > > Cc: filepro-list at lists.celestial.com
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > >
> > > Yes,
> > > In the case of *adding* characters to a file... you only need the
> > > entire disk space available (including the original file) to be the
> > > size of the "resulting" file.  so, if your file takes up 500MB and
> > > you
> >
> > > have a 600MB disk.... you can only add 100MB more to that file,
> > > however you do the math. If you take the very same 500MB file and
> > > try to "remove" characters from it... the program will break and
> > > your data
> >
> > > will be lost... and everything bad will happen, because in this case
> 
> > > you would need at least the size of the original file in free disk
> > > space *before* you start the shuffle operation.
> > >
> > > Actually, in such a situation, I think filePro's free disk free
> > > check would complain and say the re=shuffle could not be done,
> > > rather than destroy your data, but I wouldn't chance this. :-)  Do
> > > the calculating
> >
> > > yourself first... or you will end up having to break out of the
> > > define
> >
> > > program, or carefully roll black all of your changes. Yikes, it
> > > would just be a hassle.
> > >
> > > John
> > >
> > > P.s. Even though with the "in-place" re-define thing you don't need
> > > a duplicate size area of free space available as the size of your
> > > final file to do a shuffle that adds characters... I would hope that
> 
> > > you would have a few extra bytes more than the final restructured
> > > size free for safety's sake.  I don't know who safety is, but he
> > > likes extra on everything.
> > >
> > > John
> > >
> > >
> > > > -----Original Message-----
> > > > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > > > Sent: Saturday, July 26, 2008 6:38 PM
> > > > To: john at valar.com
> > > > Cc: 'filePro_List'
> > > > Subject: RE: Free Disk Necessary To Retructure A File
> > > >
> > > > John,
> > > >
> > > > So are you saying that if I had 10 million bytes free on my disk
> > > > drive, I could add a field to a file with 400mb as long as the
> > > > resulting file was not more than 10mb larger.
> > > >
> > > > In other words, I don't need free disk space to be anywhere near
> > > > the
> >
> > > > size of the file I am restructuring.
> > > >
> > > > Regards,
> > > >
> > > > Scott
> > > >
> > > >
> > > > Scott Walker
> > > > RAM Systems Corp.
> > > > ScottWalker at RAMSystemsCorp.com
> > > > Ph: (704) 896-6549
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Esak [mailto:john at valar.com]
> > > > Sent: Saturday, July 26, 2008 6:32 PM
> > > > To: 'Scott Walker'
> > > > Cc: filepro-list at lists.celestial.com
> > > > Subject: RE: Free Disk Necessary To Retructure A File
> > > >
> > > >
> > > > <top posted question about file size in re-define>
> > > >
> > > > You are right, You have plenty to spare.
> > > >
> > > > I hate working in MB's... much better to work with the actual byte
> 
> > > > size of the files and not worry about converting to millions and
> > > > then
> > >
> > > > dividing by 1024*1024, etc.  If you simply  add 20 to the 2837
> > > > (which you are heading for, (because the Define files program does
> 
> > > > not display the extra 20 byte header needed for each record),
> > > > giving
> >
> > > > you 2857 and multiply this times 159,000 + 1 (for the extra 0th
> > > > record which is also commonly forgotten... you end up with
> > > > 454265857
> >
> > > > bytes of
> > >
> > > > space needed to
> > >
> > >
> > >
> > >
> > > > hold the new file... and that is really all you need.
> > > >
> > > > Ken wrote the Define Files routine that adds characters, to do it
> > > > "in place". In other words you just move the end of file marker to
> 
> > > > the highest point the file will require after the new full size is
> 
> > > > calculated, and then writing all the records one by one from the
> > > > end
> >
> > > > of the file to their new position starting backwards from the end
> > > > of
> >
> > > > the file.  A nice, efficient way to make a file bigger back in the
> 
> > > > days when disk space was at a huge premium.
> > > >
> > > > So, you really have plenty of room for your addition.
> > > >
> > > > If you were to *shrink* the file on the other hand (take away
> > > > characters from any field), this trick is not employed so you
> > > > would need the full new size of the file in free space, at least,
> > > > for the "temp" file which would be created and then moved into
> > > > place.
> > > >
> > > > John
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: filepro-list-bounces+john=valar.com at lists.celestial.com
> > > > > [mailto:filepro-list-bounces+john=valar.com at lists.celestial.com]
> > > > > On Behalf Of Scott Walker
> > > > > Sent: Saturday, July 26, 2008 3:25 PM
> > > > > To: 'filePro_List'
> > > > > Subject: Free Disk Necessary To Retructure A File
> > > > >
> > > > > I just want to make sure...
> > > > >
> > > > > I have a key file that is 436mb.
> > > > >
> > > > > I am going to restructure it, changing the record length from
> > > > > 2735
> >
> > > > > to 2837
> > > > >
> > > > > I have 159,000 records in this file.
> > > > >
> > > > > I have 515mb of free disk space.
> > > > >
> > > > > I think I have enough disk space to do this.
> > > > >
> > > > > Am I right?
> > > > >
> > > > > Regards,
> > > > >
> > > > >
> > > > > Scott
> > > > >
> > > > >
> > > > > Scott Walker
> > > > > RAM Systems Corp.
> > > > > ScottWalker at RAMSystemsCorp.com
> > > > > Ph: (704) 896-6549
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Filepro-list mailing list Filepro-list at lists.celestial.com
> > > > > http://mailman.celestial.com/mailman/listinfo/filepro-list

More information about the Filepro-list mailing list