FW: Guru Magazine GL Article

Scott Walker ScottWalker at RAMSystemsCorp.com
Sun Jul 27 10:54:26 PDT 2008



Scott Walker
RAM Systems Corp.
ScottWalker at RAMSystemsCorp.com
Ph: (704) 896-6549




-----Original Message-----
From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com] 
Sent: Sunday, July 27, 2008 1:54 PM
To: 'john at valar.com'
Subject: RE: Guru Magazine GL Article


Yeah I loved the FPDJ also.  I told Laura I would be willing to pay many
times the price she had been charging but I guess she found that it was
just too much work.  I learn best by reading a well written article
about a topic and her journal covered several topics that I was able to
put to great use.

Regards,

Scott


Scott Walker
RAM Systems Corp.
ScottWalker at RAMSystemsCorp.com
Ph: (704) 896-6549




-----Original Message-----
From: John Esak [mailto:john at valar.com] 
Sent: Sunday, July 27, 2008 1:36 PM
To: 'Scott Walker'
Cc: filepro-list at lists.celestial.com
Subject: RE: Guru Magazine GL Article


Thanks Soott,
Yeah, too bad there isn't something like it around today... I would
actually love to see a Linux mag just like it.  And/or I have always
wanted the FPDJ back.

John

> -----Original Message-----
> From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> Sent: Sunday, July 27, 2008 1:22 PM
> To: john at valar.com
> Cc: filepro-list at lists.celestial.com
> Subject: Guru Magazine GL Article
> 
> John,
> 
> I am in the process of writing a GL Module for my application.  I just
> wanted to let you know how helpful your Guru magazine article of March

> 1991 was.  Great overview of what you need to think about before 
> starting the coding.  It's amazes me how useful that old Guru rag is 
> after all these years.
> 
> Regards,
> 
> Scott
> 
> 
> Scott Walker
> RAM Systems Corp.
> ScottWalker at RAMSystemsCorp.com
> Ph: (704) 896-6549
> 
> 
> 
> 
> -----Original Message-----
> From: John Esak [mailto:john at valar.com]
> Sent: Sunday, July 27, 2008 1:06 PM
> To: 'Scott Walker'
> Cc: filepro-list at lists.celestial.com
> Subject: RE: Free Disk Necessary To Retructure A File
> 
> 
> Oh my god... let me find a recent email about that... and I'll send it
> up.
> 
> The short answer is we had a 3Gb CPU, 300Gb Sata 3.0, 2Gb RAM system
> working on a call center app.  14 people using a FacetPhone system to 
> receive calls and store the data on the machine and each appropriate 
> scheduled time, etc. (the FacePhone being used is running on another 
> Linux server.. .unimportant to this setup.)
> 
> Anyway, the system was SO slow that almost every operation was
> noticeably slowed down... usually filePro operations, reports, etc., 
> are pretty instantaneous. On a rig like this.  However, even with no 
> one else running, it would take 27 minutes to run all the indexes on 
> the 15 files involved.
> 
> We put together a 2.4Gb CPU, 1Tb RAID 5, 2Gb RAM running Linux ubuntu.
> The system makes use of a "software" based RAID provided by the Linux.

> (Yech,,, I thought)  But, performance change was beyond staggering.  
> For example, and I swear this is true... the complete index build now 
> takes 12.12 seconds!!!!!  That is just staggering, and every other 
> operation dealing with file I/O is that much improved as well.  So, if

> you can put SATA in a RAID, preferably a true hardware RAID, not the 
> "fake" RAID some motherboards supply, you will get great performance 
> for a small group of users, like 10 to 20.  Moor than that I can't 
> verify.
> 
> As for this SATA software RAID vs a true battery-backed up hardware
> SCSI RAID a la what you would get in a new HP server, or even a 
> DELL... the performance would actually be even better (and more 
> reliable) than this. Plus, the performance would NOT degrade as you 
> add users, up to probably a couple hundred.  By the way, both machines

> in the above example are Core II Duos.  And as an aside, when we watch

> the CPU usage on the SCO side the first single drive box, it was not 
> too exciting. Only a little bit of sporadic uneven usage of both CPU's

> was noticed.  On the Linux box, the whole operation, everything, 
> system calls, file I/O, everything causes both CPU's to run almost all

> the time in a very even and smooth manner.  The Linux obviously runs 
> the threaded and both threaded programs (like filePro) in a much more 
> sophisticated way.  The downside is that with this software RAID from 
> the ubuntu driver, the CPU's are used elaborately during all file I/O 
> to the RAID.  This would be almost zero usage if we were using a good 
> battery backed up hardware adaptor for a SCSI setup.
> 
> So, to sum up, I have radically mixed apples and oranges by comparing
> RAID to a single drive and not SCSI single drive to single drive. 
> However, I wanted to make you aware that you *can* make use of these 
> inexpensive SATA drives for a filePro system with a limited number of 
> users... probably up to 50.  there are many tests showing SCSI vs SATA

> in single to single comparisons.. and they are pretty even, with SATA 
> even being faster in some instances.  This is why I tried putting our 
> application on the single SATA drive in the first place.  It was only 
> when we actually experienced the TERRIBLE slow down when all 14 users 
> were running rclerk on some high activity file I/O that it became 
> sadly very clear to me that SATA is nothing like SCSI.  A single SCSI 
> drive can easily handle any filePro app with only 14 users.  I have 
> successfully put 40 to 70 users on a single SCSI drive running filePro

> apps.... it's not my choice, but it works.  This pore little SATA 
> drive couldn't have handled even 15 users, 1 more above the 14... I 
> fear that would have broken the camel's back.  Indeed, we had some 
> pretty good size glitches even as it was.  The RAIDing of 5 drives in 
> RAID 5 fixed this all... and we didn't need to hit the big expense of 
> SCSI drives and an adaptor.  But, remember, that we will not try and 
> put more than 15 people or so on even this system.  Before too long 
> we'll build a machine that has a true SCSI or SASI RAID.  We actually 
> halved several rack mounted systems like this already, but they are 
> designated for other customers and applications, too bad.
> 
> Hope I helped you understand the somewhat subtle differences between
> SATA and SCSI.  Main difference if I didn't hit is that the SCSI 
> drives each have their own sophisticated controllers on board, and the

> adaptor/controller itself is also super sophisticated.... all of this 
> means the CPU's have to do very little work compared to their constant

> bit by bit work for a SATA drive.  This is an easily noticeable 
> difference believe me.
> 
> No time to spell check all these words... .sorry....
> 
> John
> 
> > -----Original Message-----
> > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > Sent: Saturday, July 26, 2008 8:58 PM
> > To: john at valar.com
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> > Is the performance of SCSI vastly superior over SATA in a Linux
> > server
> 
> > with multiple users?
> >
> > Scott Walker
> > RAM Systems Corp.
> > ScottWalker at RAMSystemsCorp.com
> > Ph: (704) 896-6549
> >
> >
> >
> >
> > -----Original Message-----
> > From: John Esak [mailto:john at valar.com]
> > Sent: Saturday, July 26, 2008 7:55 PM
> > To: 'Scott Walker'
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> >
> > Yeah, this is little known ... so I'm glad it helps you out.
> > However,
> 
> > customers should *always* upgrade their servers!  :-)  And,
> > seriously,
> 
> > if they are using a system with only 400MB free these days... that
> > is not enough... just on GP's.  Everyone talks about needing 10% 
> > free disk space at the very least.  I like to have 50% free whenever

> > I can get it.. .and today with 500Gb SATA drives costing $105!!!! 
> > Well, it's
> 
> > cheaper and easier to have lots of space than it ever was.  By the
> > way, I still and will probably forever support and suggest SCSI over

> > SATA, just using that as an example.
> >
> >
> > John
> >
> >
> > > -----Original Message-----
> > > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > > Sent: Saturday, July 26, 2008 7:48 PM
> > > To: john at valar.com
> > > Cc: filepro-list at lists.celestial.com
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > > John,
> > >
> > > Thank you much.  I never realized that's how it worked.  I always
> > > thought if you had a 500mb file and you were adding 100mb to it, 
> > > then you would need 1.1gb free to do it.  I thought it copied 
> > > stuff from the "old" key file to a new bigger one, and then 
> > > deleted the old key file. Thus since there would be an instant in 
> > > time where both the "old" key file existed and the new one 
> > > existed, just before
> 
> > > it deleted
> >
> > > the old one I would need enough disk space for both.
> > >
> > > This is really a case of the customer causing the problem.  I have
> > > been telling them for (3) years that they must upgrade their 
> > > server.
> > >
> > > Thanks again for the info.
> > >
> > > Regards,
> > >
> > > Scott
> > >
> > > Scott Walker
> > > RAM Systems Corp.
> > > ScottWalker at RAMSystemsCorp.com
> > > Ph: (704) 896-6549
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Esak [mailto:john at valar.com]
> > > Sent: Saturday, July 26, 2008 7:31 PM
> > > To: 'Scott Walker'
> > > Cc: filepro-list at lists.celestial.com
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > >
> > > Yes,
> > > In the case of *adding* characters to a file... you only need the
> > > entire disk space available (including the original file) to be 
> > > the size of the "resulting" file.  so, if your file takes up 500MB

> > > and you
> >
> > > have a 600MB disk.... you can only add 100MB more to that file,
> > > however you do the math. If you take the very same 500MB file and 
> > > try to "remove" characters from it... the program will break and 
> > > your data
> >
> > > will be lost... and everything bad will happen, because in this
> > > case
> 
> > > you would need at least the size of the original file in free disk
> > > space *before* you start the shuffle operation.
> > >
> > > Actually, in such a situation, I think filePro's free disk free
> > > check would complain and say the re=shuffle could not be done, 
> > > rather than destroy your data, but I wouldn't chance this. :-)  Do

> > > the calculating
> >
> > > yourself first... or you will end up having to break out of the
> > > define
> >
> > > program, or carefully roll black all of your changes. Yikes, it
> > > would just be a hassle.
> > >
> > > John
> > >
> > > P.s. Even though with the "in-place" re-define thing you don't
> > > need a duplicate size area of free space available as the size of 
> > > your final file to do a shuffle that adds characters... I would 
> > > hope that
> 
> > > you would have a few extra bytes more than the final restructured
> > > size free for safety's sake.  I don't know who safety is, but he 
> > > likes extra on everything.
> > >
> > > John
> > >
> > >
> > > > -----Original Message-----
> > > > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > > > Sent: Saturday, July 26, 2008 6:38 PM
> > > > To: john at valar.com
> > > > Cc: 'filePro_List'
> > > > Subject: RE: Free Disk Necessary To Retructure A File
> > > >
> > > > John,
> > > >
> > > > So are you saying that if I had 10 million bytes free on my disk
> > > > drive, I could add a field to a file with 400mb as long as the 
> > > > resulting file was not more than 10mb larger.
> > > >
> > > > In other words, I don't need free disk space to be anywhere near
> > > > the
> >
> > > > size of the file I am restructuring.
> > > >
> > > > Regards,
> > > >
> > > > Scott
> > > >
> > > >
> > > > Scott Walker
> > > > RAM Systems Corp.
> > > > ScottWalker at RAMSystemsCorp.com
> > > > Ph: (704) 896-6549
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Esak [mailto:john at valar.com]
> > > > Sent: Saturday, July 26, 2008 6:32 PM
> > > > To: 'Scott Walker'
> > > > Cc: filepro-list at lists.celestial.com
> > > > Subject: RE: Free Disk Necessary To Retructure A File
> > > >
> > > >
> > > > <top posted question about file size in re-define>
> > > >
> > > > You are right, You have plenty to spare.
> > > >
> > > > I hate working in MB's... much better to work with the actual
> > > > byte
> 
> > > > size of the files and not worry about converting to millions and
> > > > then
> > >
> > > > dividing by 1024*1024, etc.  If you simply  add 20 to the 2837
> > > > (which you are heading for, (because the Define files program 
> > > > does
> 
> > > > not display the extra 20 byte header needed for each record),
> > > > giving
> >
> > > > you 2857 and multiply this times 159,000 + 1 (for the extra 0th
> > > > record which is also commonly forgotten... you end up with 
> > > > 454265857
> >
> > > > bytes of
> > >
> > > > space needed to
> > >
> > >
> > >
> > >
> > > > hold the new file... and that is really all you need.
> > > >
> > > > Ken wrote the Define Files routine that adds characters, to do
> > > > it "in place". In other words you just move the end of file 
> > > > marker to
> 
> > > > the highest point the file will require after the new full size
> > > > is
> 
> > > > calculated, and then writing all the records one by one from the
> > > > end
> >
> > > > of the file to their new position starting backwards from the
> > > > end of
> >
> > > > the file.  A nice, efficient way to make a file bigger back in
> > > > the
> 
> > > > days when disk space was at a huge premium.
> > > >
> > > > So, you really have plenty of room for your addition.
> > > >
> > > > If you were to *shrink* the file on the other hand (take away
> > > > characters from any field), this trick is not employed so you 
> > > > would need the full new size of the file in free space, at 
> > > > least, for the "temp" file which would be created and then moved

> > > > into place.
> > > >
> > > > John
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: filepro-list-bounces+john=valar.com at lists.celestial.com
> > > > > [mailto:filepro-list-bounces+john=valar.com at lists.celestial.co
> > > > > m]
> > > > > On Behalf Of Scott Walker
> > > > > Sent: Saturday, July 26, 2008 3:25 PM
> > > > > To: 'filePro_List'
> > > > > Subject: Free Disk Necessary To Retructure A File
> > > > >
> > > > > I just want to make sure...
> > > > >
> > > > > I have a key file that is 436mb.
> > > > >
> > > > > I am going to restructure it, changing the record length from
> > > > > 2735
> >
> > > > > to 2837
> > > > >
> > > > > I have 159,000 records in this file.
> > > > >
> > > > > I have 515mb of free disk space.
> > > > >
> > > > > I think I have enough disk space to do this.
> > > > >
> > > > > Am I right?
> > > > >
> > > > > Regards,
> > > > >
> > > > >
> > > > > Scott
> > > > >
> > > > >
> > > > > Scott Walker
> > > > > RAM Systems Corp.
> > > > > ScottWalker at RAMSystemsCorp.com
> > > > > Ph: (704) 896-6549
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Filepro-list mailing list Filepro-list at lists.celestial.com
> > > > > http://mailman.celestial.com/mailman/listinfo/filepro-list




More information about the Filepro-list mailing list