Long running output

[Nancy Palmquist]

Richard,

On 3/4/2016 7:35 PM, Mike Schwartz via Filepro-list wrote:
>> One of my client's has an output (creates a csv file) that takes a long
> time to
>> run.  Last Saturday, on their system, it took just over 4 hours.  On
> Tuesday
>
> [Snip]
>
>> processor and more memory.  It may take just running this output directly
> on
>> the server to have it finish in a reasonable amount of time.
>> Richard Kreiss
> ----- ----- -----
>       I think you answered your own question.  Remember that 40,000 records
> per item must be dragged over your network cable from the server to the
> workstation in order to do all the calculations.  If the calculations are
> making any changes to these records, all 40,000 for each item must be
> written BACK to the Windows server over your network cable.  That is a heck
> of a lot of data to transfer over your network cables.
>
>       I've had a few customers try to convert from Unix  to MS-Windows.
> Every time it was a disaster.  One customer in Chicago used to run a 1 hour
> batch of reports in the evening.  After they converted over to a MS-Windows
> server, they started the batch from a workstation on Friday evening and the
> batch wasn't done running on Monday!
>
>       Running the batch from the Windows NT server console was a lot better.
> The reports would finish up in about 4 hours.  (Their new Windows NT server
> was 4 times slower than their old Xenix computer was...)
>
>      This was OK for the weekend reports, but users couldn't run any of the
> other reports they used to run from their workstations.  Schedules that used
> to take a minute or two to print on the old Xenix server took hours to
> generate and print via their Windows network.
>
>       As a matter of fact I gained the customer because they wanted me to
> build them a new Unix server and move them BACK to Unix.
>
>       So, your options are:
>
>       1)  Install faster network cabling and network switches
>
>       2)  Run all the long reports from the Windows Server console.
>
>       3)  Or, I guess I should add:  convert them to one of the Unixes.
>
>       I am just curious enough that, if the data is not too proprietary or
> confidential,  I wonder if I could get a dataset from you to run on one of
> my Unix servers???
>
> Mike Schwartz

I agree entirely with Mike's assessment.  The I/O and network functions 
on WINDOWS are very slow compared to any Unix/Linux system.  The 
difference between, client side and server side processing.

I have seen the same difference between running on a Windows Server - 
where the drives are local and running from any client on the network.  
The difference is huge.

To run purges on our Windows Data, we actually copy the data to a Linux 
server, run the purges and rebuild the indexes, then copy it back to 
Windows.  Otherwise, it took too long to make it reasonable.  (We have 
this luxury at that customers installation.) BTW - we just copy KEY,* 
DATA* from Windows to Linux and KEY* DATA* INDEX* back.  NEVER copy maps 
or other formats - they will not work to just copy.
( Both systems have the same version of filepro and none of the data is 
encrypted (Have not tried that))


I have used your method to greatly speed up processes.  Are you sure 
your getnext is only viewing the detail associated with the current 
record?  It is easy to have a getnext pass the end of the data and 
continue into the next set.

Also, does this data change after recorded or after some time?  It might 
make sense to add a data base that summarizes by week or something that 
would represent data that is static after a certain point.  If 40,000 
could be summarized to 100 it would greatly improve your throughput.

Nancy




> Nancy Palmquist         MOS & filePro Training Available
> Virtual Software Systems    Web Based Training and Consulting
> PHONE: (412) 835-9417           Web site:  http://www.vss3.com