FW: Guru Magazine GL Article
Scott Walker
ScottWalker at RAMSystemsCorp.com
Sun Jul 27 10:54:26 PDT 2008
Scott Walker
RAM Systems Corp.
ScottWalker at RAMSystemsCorp.com
Ph: (704) 896-6549
-----Original Message-----
From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
Sent: Sunday, July 27, 2008 1:54 PM
To: 'john at valar.com'
Subject: RE: Guru Magazine GL Article
Yeah I loved the FPDJ also. I told Laura I would be willing to pay many
times the price she had been charging but I guess she found that it was
just too much work. I learn best by reading a well written article
about a topic and her journal covered several topics that I was able to
put to great use.
Regards,
Scott
Scott Walker
RAM Systems Corp.
ScottWalker at RAMSystemsCorp.com
Ph: (704) 896-6549
-----Original Message-----
From: John Esak [mailto:john at valar.com]
Sent: Sunday, July 27, 2008 1:36 PM
To: 'Scott Walker'
Cc: filepro-list at lists.celestial.com
Subject: RE: Guru Magazine GL Article
Thanks Soott,
Yeah, too bad there isn't something like it around today... I would
actually love to see a Linux mag just like it. And/or I have always
wanted the FPDJ back.
John
> -----Original Message-----
> From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> Sent: Sunday, July 27, 2008 1:22 PM
> To: john at valar.com
> Cc: filepro-list at lists.celestial.com
> Subject: Guru Magazine GL Article
>
> John,
>
> I am in the process of writing a GL Module for my application. I just
> wanted to let you know how helpful your Guru magazine article of March
> 1991 was. Great overview of what you need to think about before
> starting the coding. It's amazes me how useful that old Guru rag is
> after all these years.
>
> Regards,
>
> Scott
>
>
> Scott Walker
> RAM Systems Corp.
> ScottWalker at RAMSystemsCorp.com
> Ph: (704) 896-6549
>
>
>
>
> -----Original Message-----
> From: John Esak [mailto:john at valar.com]
> Sent: Sunday, July 27, 2008 1:06 PM
> To: 'Scott Walker'
> Cc: filepro-list at lists.celestial.com
> Subject: RE: Free Disk Necessary To Retructure A File
>
>
> Oh my god... let me find a recent email about that... and I'll send it
> up.
>
> The short answer is we had a 3Gb CPU, 300Gb Sata 3.0, 2Gb RAM system
> working on a call center app. 14 people using a FacetPhone system to
> receive calls and store the data on the machine and each appropriate
> scheduled time, etc. (the FacePhone being used is running on another
> Linux server.. .unimportant to this setup.)
>
> Anyway, the system was SO slow that almost every operation was
> noticeably slowed down... usually filePro operations, reports, etc.,
> are pretty instantaneous. On a rig like this. However, even with no
> one else running, it would take 27 minutes to run all the indexes on
> the 15 files involved.
>
> We put together a 2.4Gb CPU, 1Tb RAID 5, 2Gb RAM running Linux ubuntu.
> The system makes use of a "software" based RAID provided by the Linux.
> (Yech,,, I thought) But, performance change was beyond staggering.
> For example, and I swear this is true... the complete index build now
> takes 12.12 seconds!!!!! That is just staggering, and every other
> operation dealing with file I/O is that much improved as well. So, if
> you can put SATA in a RAID, preferably a true hardware RAID, not the
> "fake" RAID some motherboards supply, you will get great performance
> for a small group of users, like 10 to 20. Moor than that I can't
> verify.
>
> As for this SATA software RAID vs a true battery-backed up hardware
> SCSI RAID a la what you would get in a new HP server, or even a
> DELL... the performance would actually be even better (and more
> reliable) than this. Plus, the performance would NOT degrade as you
> add users, up to probably a couple hundred. By the way, both machines
> in the above example are Core II Duos. And as an aside, when we watch
> the CPU usage on the SCO side the first single drive box, it was not
> too exciting. Only a little bit of sporadic uneven usage of both CPU's
> was noticed. On the Linux box, the whole operation, everything,
> system calls, file I/O, everything causes both CPU's to run almost all
> the time in a very even and smooth manner. The Linux obviously runs
> the threaded and both threaded programs (like filePro) in a much more
> sophisticated way. The downside is that with this software RAID from
> the ubuntu driver, the CPU's are used elaborately during all file I/O
> to the RAID. This would be almost zero usage if we were using a good
> battery backed up hardware adaptor for a SCSI setup.
>
> So, to sum up, I have radically mixed apples and oranges by comparing
> RAID to a single drive and not SCSI single drive to single drive.
> However, I wanted to make you aware that you *can* make use of these
> inexpensive SATA drives for a filePro system with a limited number of
> users... probably up to 50. there are many tests showing SCSI vs SATA
> in single to single comparisons.. and they are pretty even, with SATA
> even being faster in some instances. This is why I tried putting our
> application on the single SATA drive in the first place. It was only
> when we actually experienced the TERRIBLE slow down when all 14 users
> were running rclerk on some high activity file I/O that it became
> sadly very clear to me that SATA is nothing like SCSI. A single SCSI
> drive can easily handle any filePro app with only 14 users. I have
> successfully put 40 to 70 users on a single SCSI drive running filePro
> apps.... it's not my choice, but it works. This pore little SATA
> drive couldn't have handled even 15 users, 1 more above the 14... I
> fear that would have broken the camel's back. Indeed, we had some
> pretty good size glitches even as it was. The RAIDing of 5 drives in
> RAID 5 fixed this all... and we didn't need to hit the big expense of
> SCSI drives and an adaptor. But, remember, that we will not try and
> put more than 15 people or so on even this system. Before too long
> we'll build a machine that has a true SCSI or SASI RAID. We actually
> halved several rack mounted systems like this already, but they are
> designated for other customers and applications, too bad.
>
> Hope I helped you understand the somewhat subtle differences between
> SATA and SCSI. Main difference if I didn't hit is that the SCSI
> drives each have their own sophisticated controllers on board, and the
> adaptor/controller itself is also super sophisticated.... all of this
> means the CPU's have to do very little work compared to their constant
> bit by bit work for a SATA drive. This is an easily noticeable
> difference believe me.
>
> No time to spell check all these words... .sorry....
>
> John
>
> > -----Original Message-----
> > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > Sent: Saturday, July 26, 2008 8:58 PM
> > To: john at valar.com
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> > Is the performance of SCSI vastly superior over SATA in a Linux
> > server
>
> > with multiple users?
> >
> > Scott Walker
> > RAM Systems Corp.
> > ScottWalker at RAMSystemsCorp.com
> > Ph: (704) 896-6549
> >
> >
> >
> >
> > -----Original Message-----
> > From: John Esak [mailto:john at valar.com]
> > Sent: Saturday, July 26, 2008 7:55 PM
> > To: 'Scott Walker'
> > Cc: filepro-list at lists.celestial.com
> > Subject: RE: Free Disk Necessary To Retructure A File
> >
> >
> > Yeah, this is little known ... so I'm glad it helps you out.
> > However,
>
> > customers should *always* upgrade their servers! :-) And,
> > seriously,
>
> > if they are using a system with only 400MB free these days... that
> > is not enough... just on GP's. Everyone talks about needing 10%
> > free disk space at the very least. I like to have 50% free whenever
> > I can get it.. .and today with 500Gb SATA drives costing $105!!!!
> > Well, it's
>
> > cheaper and easier to have lots of space than it ever was. By the
> > way, I still and will probably forever support and suggest SCSI over
> > SATA, just using that as an example.
> >
> >
> > John
> >
> >
> > > -----Original Message-----
> > > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > > Sent: Saturday, July 26, 2008 7:48 PM
> > > To: john at valar.com
> > > Cc: filepro-list at lists.celestial.com
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > > John,
> > >
> > > Thank you much. I never realized that's how it worked. I always
> > > thought if you had a 500mb file and you were adding 100mb to it,
> > > then you would need 1.1gb free to do it. I thought it copied
> > > stuff from the "old" key file to a new bigger one, and then
> > > deleted the old key file. Thus since there would be an instant in
> > > time where both the "old" key file existed and the new one
> > > existed, just before
>
> > > it deleted
> >
> > > the old one I would need enough disk space for both.
> > >
> > > This is really a case of the customer causing the problem. I have
> > > been telling them for (3) years that they must upgrade their
> > > server.
> > >
> > > Thanks again for the info.
> > >
> > > Regards,
> > >
> > > Scott
> > >
> > > Scott Walker
> > > RAM Systems Corp.
> > > ScottWalker at RAMSystemsCorp.com
> > > Ph: (704) 896-6549
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Esak [mailto:john at valar.com]
> > > Sent: Saturday, July 26, 2008 7:31 PM
> > > To: 'Scott Walker'
> > > Cc: filepro-list at lists.celestial.com
> > > Subject: RE: Free Disk Necessary To Retructure A File
> > >
> > >
> > > Yes,
> > > In the case of *adding* characters to a file... you only need the
> > > entire disk space available (including the original file) to be
> > > the size of the "resulting" file. so, if your file takes up 500MB
> > > and you
> >
> > > have a 600MB disk.... you can only add 100MB more to that file,
> > > however you do the math. If you take the very same 500MB file and
> > > try to "remove" characters from it... the program will break and
> > > your data
> >
> > > will be lost... and everything bad will happen, because in this
> > > case
>
> > > you would need at least the size of the original file in free disk
> > > space *before* you start the shuffle operation.
> > >
> > > Actually, in such a situation, I think filePro's free disk free
> > > check would complain and say the re=shuffle could not be done,
> > > rather than destroy your data, but I wouldn't chance this. :-) Do
> > > the calculating
> >
> > > yourself first... or you will end up having to break out of the
> > > define
> >
> > > program, or carefully roll black all of your changes. Yikes, it
> > > would just be a hassle.
> > >
> > > John
> > >
> > > P.s. Even though with the "in-place" re-define thing you don't
> > > need a duplicate size area of free space available as the size of
> > > your final file to do a shuffle that adds characters... I would
> > > hope that
>
> > > you would have a few extra bytes more than the final restructured
> > > size free for safety's sake. I don't know who safety is, but he
> > > likes extra on everything.
> > >
> > > John
> > >
> > >
> > > > -----Original Message-----
> > > > From: Scott Walker [mailto:ScottWalker at RAMSystemsCorp.com]
> > > > Sent: Saturday, July 26, 2008 6:38 PM
> > > > To: john at valar.com
> > > > Cc: 'filePro_List'
> > > > Subject: RE: Free Disk Necessary To Retructure A File
> > > >
> > > > John,
> > > >
> > > > So are you saying that if I had 10 million bytes free on my disk
> > > > drive, I could add a field to a file with 400mb as long as the
> > > > resulting file was not more than 10mb larger.
> > > >
> > > > In other words, I don't need free disk space to be anywhere near
> > > > the
> >
> > > > size of the file I am restructuring.
> > > >
> > > > Regards,
> > > >
> > > > Scott
> > > >
> > > >
> > > > Scott Walker
> > > > RAM Systems Corp.
> > > > ScottWalker at RAMSystemsCorp.com
> > > > Ph: (704) 896-6549
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Esak [mailto:john at valar.com]
> > > > Sent: Saturday, July 26, 2008 6:32 PM
> > > > To: 'Scott Walker'
> > > > Cc: filepro-list at lists.celestial.com
> > > > Subject: RE: Free Disk Necessary To Retructure A File
> > > >
> > > >
> > > > <top posted question about file size in re-define>
> > > >
> > > > You are right, You have plenty to spare.
> > > >
> > > > I hate working in MB's... much better to work with the actual
> > > > byte
>
> > > > size of the files and not worry about converting to millions and
> > > > then
> > >
> > > > dividing by 1024*1024, etc. If you simply add 20 to the 2837
> > > > (which you are heading for, (because the Define files program
> > > > does
>
> > > > not display the extra 20 byte header needed for each record),
> > > > giving
> >
> > > > you 2857 and multiply this times 159,000 + 1 (for the extra 0th
> > > > record which is also commonly forgotten... you end up with
> > > > 454265857
> >
> > > > bytes of
> > >
> > > > space needed to
> > >
> > >
> > >
> > >
> > > > hold the new file... and that is really all you need.
> > > >
> > > > Ken wrote the Define Files routine that adds characters, to do
> > > > it "in place". In other words you just move the end of file
> > > > marker to
>
> > > > the highest point the file will require after the new full size
> > > > is
>
> > > > calculated, and then writing all the records one by one from the
> > > > end
> >
> > > > of the file to their new position starting backwards from the
> > > > end of
> >
> > > > the file. A nice, efficient way to make a file bigger back in
> > > > the
>
> > > > days when disk space was at a huge premium.
> > > >
> > > > So, you really have plenty of room for your addition.
> > > >
> > > > If you were to *shrink* the file on the other hand (take away
> > > > characters from any field), this trick is not employed so you
> > > > would need the full new size of the file in free space, at
> > > > least, for the "temp" file which would be created and then moved
> > > > into place.
> > > >
> > > > John
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: filepro-list-bounces+john=valar.com at lists.celestial.com
> > > > > [mailto:filepro-list-bounces+john=valar.com at lists.celestial.co
> > > > > m]
> > > > > On Behalf Of Scott Walker
> > > > > Sent: Saturday, July 26, 2008 3:25 PM
> > > > > To: 'filePro_List'
> > > > > Subject: Free Disk Necessary To Retructure A File
> > > > >
> > > > > I just want to make sure...
> > > > >
> > > > > I have a key file that is 436mb.
> > > > >
> > > > > I am going to restructure it, changing the record length from
> > > > > 2735
> >
> > > > > to 2837
> > > > >
> > > > > I have 159,000 records in this file.
> > > > >
> > > > > I have 515mb of free disk space.
> > > > >
> > > > > I think I have enough disk space to do this.
> > > > >
> > > > > Am I right?
> > > > >
> > > > > Regards,
> > > > >
> > > > >
> > > > > Scott
> > > > >
> > > > >
> > > > > Scott Walker
> > > > > RAM Systems Corp.
> > > > > ScottWalker at RAMSystemsCorp.com
> > > > > Ph: (704) 896-6549
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Filepro-list mailing list Filepro-list at lists.celestial.com
> > > > > http://mailman.celestial.com/mailman/listinfo/filepro-list
More information about the Filepro-list
mailing list