Long running output

[Richard Kreiss]

One of my client's has an output (creates a csv file) that takes a long time to run.  Last Saturday, on their system, it took just over 4 hours.  On Tuesday the job has to be cancelled as it was running over 6 hours and was only 1/4 of the way through.  This output is a task that runs Saturday afternoon (2 PM- no one working) and on very early on Tuesday morning( 2 AM) .  

The parameters of this output.

Process all of the active items in the item master (just over 4300 items)

As each item record is selected, do a look to the order detail file and run a set of calculations (dummy fields) for each order detail record selected.  Once all of the order detail records have been processed, export the line detail to the csv file.  Simple enough.  The problem is, on average, there are 40,000 order detail line to loop through using a getnext.  this is using an indexed lookup on the item #.

There is a posting procedure which would reduce the number or order line to process, but I can't get them to run it.

My real question is why would the output complete a about 4 hours on Saturday and run for over 6 hours on Tuesday morning.

I have disabled all of the tasks I had created which run on Monday night and early Tuesday morning.  I did the same things for Wednesday & Thursday tasks.  There are a few that still run but they take under 15 minutes to run and complete will before the report runs.

Also, this report is started remotely on the office managers computer.  We installed an SSD and the csv file is created on that drive.

Any suggestion as to what I should look for to see why the Saturday output completed and the Tuesday morning output ran too slow?

I am currently running this output on my system with my client's data (downloaded last night)/  I modified the output process slightly so that:

1. The start time is displayed
2.The end time will be displayed
3. the total run time will be displayed
4. The count of order details is displayed after each 50 records are processed.  This was to that I knew something was being processed; otherwise it looked like  the program had hung up.

My workstation is running Windows 10 (64bit) with an intel core I5-4690K 3.56Ghz, 16GB RAM and outputting to a 256GB SSD.  Server Window server 2008 R2 4GB memory, Network 1GB. FilePro 5.8 64bit.

Roughly speaking, this output is processing some 175,480,000 records. (4387*40,000).

I stated this run at 1:06 PM and it has processed 1425 item records or 57,000,000 records.  Based on a quick calculation, 4.3 item records a minute are being processed. By 6:15PM

My system is not running any background program.  No Norton on this system.

Short of running the posting remotely, any suggestion, in programming I could use to speed thing up.

Within the next few weeks we will be installing a new server with a faster processor and more memory.  It may take just running this output directly on the server to have it finish in a reasonable amount of time.

Thanks for any suggestions.

Richard Kreiss
GCC Consulting