Guru Magazine GL Article
Scott Walker
ScottWalker at RAMSystemsCorp.com
Sun Jul 27 10:21:58 PDT 2008
John,
I am in the process of writing a GL Module for my application. I just
wanted to let you know how helpful your Guru magazine article of March
1991 was. Great overview of what you need to think about before
starting the coding. It's amazes me how useful that old Guru rag is
after all these years.
Regards,
Scott
Scott Walker
RAM Systems Corp.
ScottWalker at RAMSystemsCorp.com
Ph: (704) 896-6549
-----Original Message-----
From: John Esak [mailto:john at valar.com]
Sent: Sunday, July 27, 2008 1:06 PM
To: 'Scott Walker'
Cc: filepro-list at lists.celestial.com
Subject: RE: Free Disk Necessary To Retructure A File
Oh my god... let me find a recent email about that... and I'll send it
up.
The short answer is we had a 3Gb CPU, 300Gb Sata 3.0, 2Gb RAM system
working on a call center app. 14 people using a FacetPhone system to
receive calls and store the data on the machine and each appropriate
scheduled time, etc. (the FacePhone being used is running on another
Linux server.. .unimportant to this setup.)
Anyway, the system was SO slow that almost every operation was
noticeably slowed down... usually filePro operations, reports, etc., are
pretty instantaneous. On a rig like this. However, even with no one
else running, it would take 27 minutes to run all the indexes on the 15
files involved.
We put together a 2.4Gb CPU, 1Tb RAID 5, 2Gb RAM running Linux ubuntu.
The system makes use of a "software" based RAID provided by the Linux.
(Yech,,, I thought) But, performance change was beyond staggering. For
example, and I swear this is true... the complete index build now takes
12.12 seconds!!!!! That is just staggering, and every other operation
dealing with file I/O is that much improved as well. So, if you can put
SATA in a RAID, preferably a true hardware RAID, not the "fake" RAID
some motherboards supply, you will get great performance for a small
group of users, like 10 to 20. Moor than that I can't verify.
As for this SATA software RAID vs a true battery-backed up hardware SCSI
RAID a la what you would get in a new HP server, or even a DELL... the
performance would actually be even better (and more reliable) than this.
Plus, the performance would NOT degrade as you add users, up to probably
a couple hundred. By the way, both machines in the above example are
Core II Duos. And as an aside, when we watch the CPU usage on the SCO
side the first single drive box, it was not too exciting. Only a little
bit of sporadic uneven usage of both CPU's was noticed. On the Linux
box, the whole operation, everything, system calls, file I/O, everything
causes both CPU's to run almost all the time in a very even and smooth
manner. The Linux obviously runs the threaded and both threaded
programs (like filePro) in a much more sophisticated way. The downside
is that with this software RAID from the ubuntu driver, the CPU's are
used elaborately during all file I/O to the RAID. This would be almost
zero usage if we were using a good battery backed up hardware adaptor
for a SCSI setup.
So, to sum up, I have radically mixed apples and oranges by comparing
RAID to a single drive and not SCSI single drive to single drive.
However, I wanted to make you aware that you *can* make use of these
inexpensive SATA drives for a filePro system with a limited number of
users... probably up to 50. there are many tests showing SCSI vs SATA
in single to single comparisons.. and they are pretty even, with SATA
even being faster in some instances. This is why I tried putting our
application on the single SATA drive in the first place. It was only
when we actually experienced the TERRIBLE slow down when all 14 users
were running rclerk on some high activity file I/O that it became sadly
very clear to me that SATA is nothing like SCSI. A single SCSI drive
can easily handle any filePro app with only 14 users. I have
successfully put 40 to 70 users on a single SCSI drive running filePro
apps.... it's not my choice, but it works. This pore little SATA drive
couldn't have handled even 15 users, 1 more above the 14... I fear that
would have broken the camel's back. Indeed, we had some pretty good
size glitches even as it was. The RAIDing of 5 drives in RAID 5 fixed
this all... and we didn't need to hit the big expense of SCSI drives and
an adaptor. But, remember, that we will not try and put more than 15
people or so on even this system. Before too long we'll build a machine
that has a true SCSI or SASI RAID. We actually halved several rack
mounted systems like this already, but they are designated for other
customers and applications, too bad.
Hope I helped you understand the somewhat subtle differences between
SATA and SCSI. Main difference if I didn't hit is that the SCSI drives
each have their own sophisticated controllers on board, and the
adaptor/controller itself is also super sophisticated.... all of this
means the CPU's have to do very little work compared to their constant
bit by bit work for a SATA drive. This is an easily noticeable
difference believe me.
No time to spell check all these words... .sorry....
John
> -----Original Message-----
> From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> Sent: Saturday, July 26, 2008 8:58 PM
> To: john at valar.com
> Cc: filepro-list at lists.celestial.com
> Subject: RE: Free Disk Necessary To Retructure A File
>
> Is the performance of SCSI vastly superior over SATA in a Linux server
> with multiple users?
>
> Scott Walker
> RAM Systems Corp.
> ScottWalker at RAMSystemsCorp.com
> Ph: (704) 896-6549
>
>
>
>
> -----Original Message-----
> From: John Esak [mailto:john at valar.com]
> Sent: Saturday, July 26, 2008 7:55 PM
> To: 'Scott Walker'
> Cc: filepro-list at lists.celestial.com
> Subject: RE: Free Disk Necessary To Retructure A File
>
>
> Yeah, this is little known ... so I'm glad it helps you out. However,
> customers should *always* upgrade their servers! :-) And, seriously,
> if they are using a system with only 400MB free these days... that is
> not enough... just on GP's. Everyone talks about needing 10% free
> disk space at the very least. I like to have 50% free whenever I can
> get it.. .and today with 500Gb SATA drives costing $105!!!! Well, it's
> cheaper and easier to have lots of space than it ever was. By the
> way, I still and will probably forever support and suggest SCSI over
> SATA, just using that as an example.
>
>
> John
>
>
> > -----Original Message-----
> > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > Sent: Saturday, July 26, 2008 7:48 PM
> > To: john at valar.com
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> > John,
> >
> > Thank you much. I never realized that's how it worked. I always
> > thought if you had a 500mb file and you were adding 100mb to it,
> > then you would need 1.1gb free to do it. I thought it copied stuff
> > from the "old" key file to a new bigger one, and then deleted the
> > old key file. Thus since there would be an instant in time where
> > both the "old" key file existed and the new one existed, just before
> > it deleted
>
> > the old one I would need enough disk space for both.
> >
> > This is really a case of the customer causing the problem. I have
> > been telling them for (3) years that they must upgrade their server.
> >
> > Thanks again for the info.
> >
> > Regards,
> >
> > Scott
> >
> > Scott Walker
> > RAM Systems Corp.
> > ScottWalker at RAMSystemsCorp.com
> > Ph: (704) 896-6549
> >
> >
> >
> >
> > -----Original Message-----
> > From: John Esak [mailto:john at valar.com]
> > Sent: Saturday, July 26, 2008 7:31 PM
> > To: 'Scott Walker'
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> >
> > Yes,
> > In the case of *adding* characters to a file... you only need the
> > entire disk space available (including the original file) to be the
> > size of the "resulting" file. so, if your file takes up 500MB and
> > you
>
> > have a 600MB disk.... you can only add 100MB more to that file,
> > however you do the math. If you take the very same 500MB file and
> > try to "remove" characters from it... the program will break and
> > your data
>
> > will be lost... and everything bad will happen, because in this case
> > you would need at least the size of the original file in free disk
> > space *before* you start the shuffle operation.
> >
> > Actually, in such a situation, I think filePro's free disk free
> > check would complain and say the re=shuffle could not be done,
> > rather than destroy your data, but I wouldn't chance this. :-) Do
> > the calculating
>
> > yourself first... or you will end up having to break out of the
> > define
>
> > program, or carefully roll black all of your changes. Yikes, it
> > would just be a hassle.
> >
> > John
> >
> > P.s. Even though with the "in-place" re-define thing you don't need
> > a duplicate size area of free space available as the size of your
> > final file to do a shuffle that adds characters... I would hope that
> > you would have a few extra bytes more than the final restructured
> > size free for safety's sake. I don't know who safety is, but he
> > likes extra on everything.
> >
> > John
> >
> >
> > > -----Original Message-----
> > > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > > Sent: Saturday, July 26, 2008 6:38 PM
> > > To: john at valar.com
> > > Cc: 'filePro_List'
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > > John,
> > >
> > > So are you saying that if I had 10 million bytes free on my disk
> > > drive, I could add a field to a file with 400mb as long as the
> > > resulting file was not more than 10mb larger.
> > >
> > > In other words, I don't need free disk space to be anywhere near
> > > the
>
> > > size of the file I am restructuring.
> > >
> > > Regards,
> > >
> > > Scott
> > >
> > >
> > > Scott Walker
> > > RAM Systems Corp.
> > > ScottWalker at RAMSystemsCorp.com
> > > Ph: (704) 896-6549
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Esak [mailto:john at valar.com]
> > > Sent: Saturday, July 26, 2008 6:32 PM
> > > To: 'Scott Walker'
> > > Cc: filepro-list at lists.celestial.com
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > >
> > > <top posted question about file size in re-define>
> > >
> > > You are right, You have plenty to spare.
> > >
> > > I hate working in MB's... much better to work with the actual byte
> > > size of the files and not worry about converting to millions and
> > > then
> >
> > > dividing by 1024*1024, etc. If you simply add 20 to the 2837
> > > (which you are heading for, (because the Define files program does
> > > not display the extra 20 byte header needed for each record),
> > > giving
>
> > > you 2857 and multiply this times 159,000 + 1 (for the extra 0th
> > > record which is also commonly forgotten... you end up with
> > > 454265857
>
> > > bytes of
> >
> > > space needed to
> >
> >
> >
> >
> > > hold the new file... and that is really all you need.
> > >
> > > Ken wrote the Define Files routine that adds characters, to do it
> > > "in place". In other words you just move the end of file marker to
> > > the highest point the file will require after the new full size is
> > > calculated, and then writing all the records one by one from the
> > > end
>
> > > of the file to their new position starting backwards from the end
> > > of
>
> > > the file. A nice, efficient way to make a file bigger back in the
> > > days when disk space was at a huge premium.
> > >
> > > So, you really have plenty of room for your addition.
> > >
> > > If you were to *shrink* the file on the other hand (take away
> > > characters from any field), this trick is not employed so you
> > > would need the full new size of the file in free space, at least,
> > > for the "temp" file which would be created and then moved into
> > > place.
> > >
> > > John
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: filepro-list-bounces+john=valar.com at lists.celestial.com
> > > > [mailto:filepro-list-bounces+john=valar.com at lists.celestial.com]
> > > > On Behalf Of Scott Walker
> > > > Sent: Saturday, July 26, 2008 3:25 PM
> > > > To: 'filePro_List'
> > > > Subject: Free Disk Necessary To Retructure A File
> > > >
> > > > I just want to make sure...
> > > >
> > > > I have a key file that is 436mb.
> > > >
> > > > I am going to restructure it, changing the record length from
> > > > 2735
>
> > > > to 2837
> > > >
> > > > I have 159,000 records in this file.
> > > >
> > > > I have 515mb of free disk space.
> > > >
> > > > I think I have enough disk space to do this.
> > > >
> > > > Am I right?
> > > >
> > > > Regards,
> > > >
> > > >
> > > > Scott
> > > >
> > > >
> > > > Scott Walker
> > > > RAM Systems Corp.
> > > > ScottWalker at RAMSystemsCorp.com
> > > > Ph: (704) 896-6549
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Filepro-list mailing list Filepro-list at lists.celestial.com
> > > > http://mailman.celestial.com/mailman/listinfo/filepro-list
More information about the Filepro-list
mailing list