Multiple autoshuf processes running concurrently

[Don Coleman]

> -----Original Message-----
> From: Nancy Palmquist [mailto:nlp at vss3.com]
> Sent: Friday, May 18, 2007 10:10 AM
> To: Don Coleman
> Cc: filepro-list at lists.celestial.com
> Subject: Re: Multiple autoshuf processes running concurrently
> 
> Don Coleman wrote:
> > Can anyone tell me if I can have multiple autoshuf processes running on
> > separate/different files at the same time, or is this a no-no?  There
> are
> > two large archive files I would like to update this weekend.  One has 22
> > qualified keys totaling 4.6 GB and the other has 22 qualified keys
> totaling
> > 26.8 GB.  I'm hoping I can run both concurrently to limit/reduce the
> time
> > their fP application is unavailable.  This client is a 24X7 operation.
> I
> > will be running these on a remote session connected to the server with
> the
> > network share disabled.
> >
> > Example, running concurrently
> > Autoshuf file_123
> > Autoshuf file_456
> >
> > Windows 2000 Advanced Server, Windows 2000 & XP Pro clients, fP v5.0.13.
> >
> > Don Coleman
> > Donald G. Coleman, Consultant
> > 402 Andrew Circle
> > Indiana, PA 15701
> > dcoleman at dgcreact.com
> > (724) 349-6302
> >
> >
> >
> > _______________________________________________
> > Filepro-list mailing list
> > Filepro-list at lists.celestial.com
> > http://mailman.celestial.com/mailman/listinfo/filepro-list
> >
> >
> Don,
> 
> It should run just fine.  But I would add that since this is a high
> resource function, it may take just as much time to run concurrent as to
> run sequenially.
> 
> Things like removing indexes and then rebuilding after, can have a huge
> impact on the speed.
> 
> I would also note that if you can do this on a *NIX based system instead
> of a WIN based the speed is hugely improved.
> 
> You need to have a map/edits converted for the platform change, but the
> key and index files will copy without alteration.
> 
> The time it might take to copy this much data from a Win to a *NIX
> system is more than made up by the speed with which it will process on
> *NIX.
> 
> My customer with large files will always do this stuff on Linux, but his
> daily work is all on WIndows2003.
> 
> We move the keys to the Linux system, last weekend we shrunk the file
> which is a fresh rewrite of every record  to a qualifier or back to the
> live system.  This also moved related records in 4 other files at the
> same time.  It was 3.5 gig of data in the working file and finished in
> less than 4 hours.  This included building A-M indexes on 10 qualifiers
> (most bigger than 1 gig) and 3 indexes on each of 30 other qualified
> files.
> 
> If you have a critical type of process like this and want good I/O and
> process speed, try to do it on Unix or Linux.
> 
> Nancy
> 
> --
> Nancy Palmquist 		MOS & filePro Training Available
> Virtual Software Systems	Web Based Training and Consulting
> PHONE: (412) 835-9417		   Web site:  http://www.vss3.com


Nancy:

I was worried about the amount of time this would take so I ran two test
yesterday.  I needed to provide an estimated time when users would again be
allowed to access the application after restructuring and other
modifications for scheduling purposes.  I was kind of surprised.  I took 2
keys from each of the two files (along with the maps and data segments) and
copied them to a separate folder on the same server.  This represented
approximately 1/11 of the data to be restructured.  The smaller of the two
files took 7 minutes to restructure and that was with me doing it from a
client with all users accessing their application in their normal path; in
other words under load.  7 minutes * 11 = 77 minutes.  I was expecting 3-5
hours.  The larger file took 21.5 minutes * 11 = 236.5 minutes or
approximately 4 hours.  When I do it for real this weekend I will be
connected to the server via a remote admin connection with no other users
connected to this server at all (network share disabled).  Looks like I may
get some sleep Saturday night after all (while the larger file restructures.

Thanks for the Unix/Linux tip.  Good to know.



Don Coleman
Donald G. Coleman, Consultant
402 Andrew Circle
Indiana, PA 15701
dcoleman at dgcreact.com
(724) 349-6302