Free Disk Necessary To Retructure A File

[John Esak]

Oh my god... let me find a recent email about that... and I'll send it up.

The short answer is we had a 3Gb CPU, 300Gb Sata 3.0, 2Gb RAM system working
on a call center app.  14 people using a FacetPhone system to receive calls
and store the data on the machine and each appropriate scheduled time, etc.
(the FacePhone being used is running on another Linux server.. .unimportant
to this setup.)  

Anyway, the system was SO slow that almost every operation was noticeably
slowed down... usually filePro operations, reports, etc., are pretty
instantaneous. On a rig like this.  However, even with no one else running,
it would take 27 minutes to run all the indexes on the 15 files involved.  

We put together a 2.4Gb CPU, 1Tb RAID 5, 2Gb RAM running Linux ubuntu.  The
system makes use of a "software" based RAID provided by the Linux.  (Yech,,,
I thought)  But, performance change was beyond staggering.  For example, and
I swear this is true... the complete index build now takes 12.12
seconds!!!!!  That is just staggering, and every other operation dealing
with file I/O is that much improved as well.  So, if you can put SATA in a
RAID, preferably a true hardware RAID, not the "fake" RAID some motherboards
supply, you will get great performance for a small group of users, like 10
to 20.  Moor than that I can't verify.

As for this SATA software RAID vs a true battery-backed up hardware SCSI
RAID a la what you would get in a new HP server, or even a DELL... the
performance would actually be even better (and more reliable) than this.
Plus, the performance would NOT degrade as you add users, up to probably a
couple hundred.  By the way, both machines in the above example are Core II
Duos.  And as an aside, when we watch the CPU usage on the SCO side the
first single drive box, it was not too exciting. Only a little bit of
sporadic uneven usage of both CPU's was noticed.  On the Linux box, the
whole operation, everything, system calls, file I/O, everything causes both
CPU's to run almost all the time in a very even and smooth manner.  The
Linux obviously runs the threaded and both threaded programs (like filePro)
in a much more sophisticated way.  The downside is that with this software
RAID from the ubuntu driver, the CPU's are used elaborately during all file
I/O to the RAID.  This would be almost zero usage if we were using a good
battery backed up hardware adaptor for a SCSI setup. 

So, to sum up, I have radically mixed apples and oranges by comparing RAID
to a single drive and not SCSI single drive to single drive.  However, I
wanted to make you aware that you *can* make use of these inexpensive SATA
drives for a filePro system with a limited number of users... probably up to
50.  there are many tests showing SCSI vs SATA in single to single
comparisons.. and they are pretty even, with SATA even being faster in some
instances.  This is why I tried putting our application on the single SATA
drive in the first place.  It was only when we actually experienced the
TERRIBLE slow down when all 14 users were running rclerk on some high
activity file I/O that it became sadly very clear to me that SATA is nothing
like SCSI.  A single SCSI drive can easily handle any filePro app with only
14 users.  I have successfully put 40 to 70 users on a single SCSI drive
running filePro apps.... it's not my choice, but it works.  This pore little
SATA drive couldn't have handled even 15 users, 1 more above the 14... I
fear that would have broken the camel's back.  Indeed, we had some pretty
good size glitches even as it was.  The RAIDing of 5 drives in RAID 5 fixed
this all... and we didn't need to hit the big expense of SCSI drives and an
adaptor.  But, remember, that we will not try and put more than 15 people or
so on even this system.  Before too long we'll build a machine that has a
true SCSI or SASI RAID.  We actually halved several rack mounted systems
like this already, but they are designated for other customers and
applications, too bad.

Hope I helped you understand the somewhat subtle differences between SATA
and SCSI.  Main difference if I didn't hit is that the SCSI drives each have
their own sophisticated controllers on board, and the adaptor/controller
itself is also super sophisticated.... all of this means the CPU's have to
do very little work compared to their constant bit by bit work for a SATA
drive.  This is an easily noticeable difference believe me.

No time to spell check all these words... .sorry....

John

> -----Original Message-----
> From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> Sent: Saturday, July 26, 2008 8:58 PM
> To: john at valar.com
> Cc: filepro-list at lists.celestial.com
> Subject: RE: Free Disk Necessary To Retructure A File
> 
> Is the performance of SCSI vastly superior over SATA in a Linux server
> with multiple users?
> 
> Scott Walker
> RAM Systems Corp.
> ScottWalker at RAMSystemsCorp.com
> Ph: (704) 896-6549
> 
> 
> 
> 
> -----Original Message-----
> From: John Esak [mailto:john at valar.com]
> Sent: Saturday, July 26, 2008 7:55 PM
> To: 'Scott Walker'
> Cc: filepro-list at lists.celestial.com
> Subject: RE: Free Disk Necessary To Retructure A File
> 
> 
> Yeah, this is little known ... so I'm glad it helps you out.  However,
> customers should *always* upgrade their servers!  :-)  And, seriously,
> if they are using a system with only 400MB free these days... that is
> not enough... just on GP's.  Everyone talks about needing 10% free disk
> space at the very least.  I like to have 50% free whenever I can get it..
> .and today with 500Gb SATA drives costing $105!!!! Well, it's cheaper
> and easier to have lots of space than it ever was.  By the way, I still
> and will probably forever support and suggest SCSI over SATA, just using
> that as an example.
> 
> 
> John
> 
> 
> > -----Original Message-----
> > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > Sent: Saturday, July 26, 2008 7:48 PM
> > To: john at valar.com
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> > John,
> >
> > Thank you much.  I never realized that's how it worked.  I always
> > thought if you had a 500mb file and you were adding 100mb to it, then
> > you would need 1.1gb free to do it.  I thought it copied stuff from
> > the "old" key file to a new bigger one, and then deleted the old key
> > file. Thus since there would be an instant in time where both the
> > "old" key file existed and the new one existed, just before it deleted
> 
> > the old one I would need enough disk space for both.
> >
> > This is really a case of the customer causing the problem.  I have
> > been telling them for (3) years that they must upgrade their server.
> >
> > Thanks again for the info.
> >
> > Regards,
> >
> > Scott
> >
> > Scott Walker
> > RAM Systems Corp.
> > ScottWalker at RAMSystemsCorp.com
> > Ph: (704) 896-6549
> >
> >
> >
> >
> > -----Original Message-----
> > From: John Esak [mailto:john at valar.com]
> > Sent: Saturday, July 26, 2008 7:31 PM
> > To: 'Scott Walker'
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> >
> > Yes,
> > In the case of *adding* characters to a file... you only need the
> > entire disk space available (including the original file) to be the
> > size of the "resulting" file.  so, if your file takes up 500MB and you
> 
> > have a 600MB disk.... you can only add 100MB more to that file,
> > however you do the math. If you take the very same 500MB file and try
> > to "remove" characters from it... the program will break and your data
> 
> > will be lost... and everything bad will happen, because in this case
> > you would need at least the size of the original file in free disk
> > space *before* you start the shuffle operation.
> >
> > Actually, in such a situation, I think filePro's free disk free check
> > would complain and say the re=shuffle could not be done, rather than
> > destroy your data, but I wouldn't chance this. :-)  Do the calculating
> 
> > yourself first... or you will end up having to break out of the define
> 
> > program, or carefully roll black all of your changes. Yikes, it would
> > just be a hassle.
> >
> > John
> >
> > P.s. Even though with the "in-place" re-define thing you don't need a
> > duplicate size area of free space available as the size of your final
> > file to do a shuffle that adds characters... I would hope that you
> > would have a few extra bytes more than the final restructured size
> > free for safety's sake.  I don't know who safety is, but he likes
> > extra on everything.
> >
> > John
> >
> >
> > > -----Original Message-----
> > > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > > Sent: Saturday, July 26, 2008 6:38 PM
> > > To: john at valar.com
> > > Cc: 'filePro_List'
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > > John,
> > >
> > > So are you saying that if I had 10 million bytes free on my disk
> > > drive, I could add a field to a file with 400mb as long as the
> > > resulting file was not more than 10mb larger.
> > >
> > > In other words, I don't need free disk space to be anywhere near the
> 
> > > size of the file I am restructuring.
> > >
> > > Regards,
> > >
> > > Scott
> > >
> > >
> > > Scott Walker
> > > RAM Systems Corp.
> > > ScottWalker at RAMSystemsCorp.com
> > > Ph: (704) 896-6549
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Esak [mailto:john at valar.com]
> > > Sent: Saturday, July 26, 2008 6:32 PM
> > > To: 'Scott Walker'
> > > Cc: filepro-list at lists.celestial.com
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > >
> > > <top posted question about file size in re-define>
> > >
> > > You are right, You have plenty to spare.
> > >
> > > I hate working in MB's... much better to work with the actual byte
> > > size of the files and not worry about converting to millions and
> > > then
> >
> > > dividing by 1024*1024, etc.  If you simply  add 20 to the 2837
> > > (which you are heading for, (because the Define files program does
> > > not display the extra 20 byte header needed for each record), giving
> 
> > > you 2857 and multiply this times 159,000 + 1 (for the extra 0th
> > > record which is also commonly forgotten... you end up with 454265857
> 
> > > bytes of
> >
> > > space needed to
> >
> >
> >
> >
> > > hold the new file... and that is really all you need.
> > >
> > > Ken wrote the Define Files routine that adds characters, to do it
> > > "in place". In other words you just move the end of file marker to
> > > the highest point the file will require after the new full size is
> > > calculated, and then writing all the records one by one from the end
> 
> > > of the file to their new position starting backwards from the end of
> 
> > > the file.  A nice, efficient way to make a file bigger back in the
> > > days when disk space was at a huge premium.
> > >
> > > So, you really have plenty of room for your addition.
> > >
> > > If you were to *shrink* the file on the other hand (take away
> > > characters from any field), this trick is not employed so you would
> > > need the full new size of the file in free space, at least, for the
> > > "temp" file which would be created and then moved into place.
> > >
> > > John
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: filepro-list-bounces+john=valar.com at lists.celestial.com
> > > > [mailto:filepro-list-bounces+john=valar.com at lists.celestial.com]
> > > > On Behalf Of Scott Walker
> > > > Sent: Saturday, July 26, 2008 3:25 PM
> > > > To: 'filePro_List'
> > > > Subject: Free Disk Necessary To Retructure A File
> > > >
> > > > I just want to make sure...
> > > >
> > > > I have a key file that is 436mb.
> > > >
> > > > I am going to restructure it, changing the record length from 2735
> 
> > > > to 2837
> > > >
> > > > I have 159,000 records in this file.
> > > >
> > > > I have 515mb of free disk space.
> > > >
> > > > I think I have enough disk space to do this.
> > > >
> > > > Am I right?
> > > >
> > > > Regards,
> > > >
> > > >
> > > > Scott
> > > >
> > > >
> > > > Scott Walker
> > > > RAM Systems Corp.
> > > > ScottWalker at RAMSystemsCorp.com
> > > > Ph: (704) 896-6549
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Filepro-list mailing list Filepro-list at lists.celestial.com
> > > > http://mailman.celestial.com/mailman/listinfo/filepro-list