limits on called program's variable and/or array space?

Mon Jul 16 06:39:34 PDT 2018

Good questions and observations, Richard.  I believe #1 to be true, but 
I can only offer an additional question related to #2.

When saving a processing table using rcabe, after answering 'Y' to 
'Check Syntax', the token table size is listed.  Can that number be 
trusted to aid in setting the appropriate *TOKSIZE value (setting the 
value to something just higher than what is listed)?

Bruce

On 7/16/18 9:00 AM, Richard Veith via Filepro-list wrote:
> I appreciated all the comments and suggestions, but after much testing I discovered that there was one variable in the called program that did not have a declared size (even though I thought I had taken  care of that), and fixing that resolved the problem.  None of the other suggestions would get the programs to run successfully without that one change.  (As noted in the original post, there is a calling program and a called program, and it was the called program that was causing the abort when the pair of programs was run on all records in a 22 million record database, producing only a Windows 7 message saying "dreport.exe has stopped working" when about 80% through the database.)
>
> But I still have some questions:
> 1.  Is PFFORMTOKSIZE the only env variable that affects the size of space available for a called program's variables?
> 2.  If so, is there an upper limit other than 999,999 in FilePro 5.7.00D9? (Because my varying the size from 200000 up to 990000 had no impact).
> 3.  Do the calling programs and called programs share a limited amount -- and the same overall limited amount -- of FilePro-reserved memory regardless of overall RAM?   And does more reserved for the calling program mean less reserved for the called program?   (On our standalone PC with 16 GB of RAM and no other user programs running, there was always about 12 GB of free RAM whether these two FilePro programs were going to abort or not.)
> 3.  One of my tests was to change the two arrays (not mapped to any record fields) in the called program from local to global, making sure I clear them at the start of each record's processing.  This did not affect the abort/no-abort issue, but it had a big impact on processing time.   When the arrays were NOT global it added over an hour to overall processing time.  Is that because the arrays get set up once, rather than 22 million times?  Would the same thing be true for dummy variables?
>
> Thanks again for any comments or suggestions.
> Rich
>
> -----Original Message-----
> From: Richard Kreiss [mailto:rkreiss at verizon.net]
> Sent: Friday, July 06, 2018 5:14 PM
> To: Richard Veith <richard.veith at smrresearch.com>
> Subject: RE: limits on called program's variable and/or array space?
>
> I realize that many have responded earlier. But one thing came to mind this afternoon, have you considered that there may be in issue with disk space.
>
> filePro creates a temp file with data for the output.  I have some reports with 8 sort levels which can create a fairly large temp file.  The size depends on the size of the fields being indexed.  Just take a look at the size of indexes.
>
> If ACL is being run on a Windows machine, one variable is the amount of space and user can use,  this may be causing an issue if the limit is reached.
>
> Just something more to think about.
>
>
> Richard Kreiss
> GCC Consulting
>
>
>
>> -----Original Message-----
>> From: Filepro-list [mailto:filepro-list-
>> bounces+rkreiss=verizon.net at lists.celestial.com] On Behalf Of Richard
> Veith via
>> Filepro-list
>> Sent: Tuesday, July 3, 2018 8:34 AM
>> To: Filepro-list at lists.celestial.com
>> Subject: RE: limits on called program's variable and/or array space?
>>
>> Thanks to those who replied.  Before responding to some of the individual
>> questions, there is a little more I could add to the story.   First, it
> takes about 6
>> hrs of processing before the abort, so testing various things takes time.
> If we
>> use smaller sets of data, it does not abort.   And second, there is also a
> Sort
>> involved for the calling program, maintained in the program's "out" file.
> It is a 3-
>> field sort with a total of 41 chars per record, for all 22 million
> records.
>> Here are some of the questions from the people who responded:
>>
>>>   it would help to know if you get an error of sort and/or if
>>> anything is logged in event logs
>> When the abort happens, Windows puts up a message "dreport.exe has
>> stopped working".  And the FilePro screen behind it shows the number
>> of records selected, and the number for "generating output".  The
>> latter number is
> never
>> the same but always seems to be within 20,000 or 30,000 of the
>> previous
> abort
>> points (and sometimes a smaller gap of just 5 or 6 records).  If there
>> is
> some
>> "event log" we can look at (either Windows or FilePro), please point
>> us in
> the
>> right direction.
>>
>>> Have you tried  a much higher PFFORMTOKSIZE and PFTOKSIZE?
>> How high can we go?
>>
>>> Are you using an auto table?
>> No.
>>
>>> Have you considered adding this as a sub routine rather the a called
> program?
>> Yes, but there are one or two other points in our entire procedure
>> where
> the
>> called program's functionality is needed, and the intent was to use
>> the
> same
>> called program in the various places.  Also wanted to keep the
>> variables separate between the calling program and the called program.
>>
>>>   If you are updating records and in these updates an index key is
> changed,
>> make sure you are locking your lookups.
>> This is a standalone PC doing only one thing at a time, and nobody
>> else is
> doing
>> any updating.  Both the calling program and the called program do
>> update records but nothing that would change an index.  And the two
>> indexes
> involved
>> are demand indexes.
>>
>>> Have you tried running the process using both dreport and rreport?
>> Have not tried rreport.  Anything in particular to look for?
>>
>> I will let you all know how this gets resolved, but as I said, it
>> takes 6
> hours or so
>> before hitting the abort, so it is a slow testing process.  And I will
>> be
> away for
>> the 4th and the rest of the week.
>>
>> Thanks again for any insights.
>> Rich
>>
>> -----Original Message-----
>> From: Richard Kreiss [mailto:rkreiss at verizon.net]
>> Sent: Monday, July 02, 2018 6:43 PM
>> To: Richard Veith <richard.veith at smrresearch.com>
>> Subject: Re: limits on called program's variable and/or array space?
>>
>> Have you considered adding this as a sub routine rather the a called
> program?
>> Richard
>> Sent from my iPhone
>>
>>> On Jul 2, 2018, at 5:58 PM, Richard Veith via Filepro-list <filepro-
>> list at lists.celestial.com> wrote:
>>> We have a database of about 20 million records, where each record is
> just
>> over 2200 characters.  As part of our processing, we run an output
>> process
> table
>> on the entire database, and this process calls another output process
> table.  The
>> calling process is about 1500 lines, and the called process is about
>> 300
> lines.
>>> Up until a recent change, this all worked fine.  The change was to
>>> add a
>> subroutine to the called process, and this subroutine included a few
>> more variables (declared with sizes, as were the previous variables)
>> as well as
> two
>> arrays.  One of the arrays is 700 elements consisting of 4 digit
>> integers,
> and the
>> other array is 700 elements consisting of 20 character strings.  The
> arrays are
>> not mapped to any existing fields.
>>> The puzzling thing is that if the called program, with the new
> subroutine, is run
>> by itself as a standalone program against the entire database, it
>> works
> fine.  But
>> when being called from within the calling program, it aborts about
>> 85% of
> the
>> way through the database.
>>> Since the calling program/called program arrangement worked fine
>>> before
> the
>> new subroutine was added, and the called program works fine on its
>> own,
> our
>> suspicion is that there is some limit on the memory allocated for
>> called
> program
>> variables and arrays, and we exceeded that.  Or those arrays are not
>> being treated as dummy variables and instead, as each record is
>> processed, a new version of each array is created and all prior ones
>> are still hanging
> around.
>>> Do you think it is a memory issue?  Is there a way to specify more
> memory for
>> the called program (we have PFFORMTOKSIZE set to 200000 on the
>> assumption it might be relevant since the documentation for CALL says
>> "the called processing table uses the token area reserved for printing
>> a form from
> Inquire,
>> Update, Add")?  Is there a way to remove the arrays as processing
>> finishes
> one
>> record and moves to the next record? (We are using "clear" on the
>> arrays
> but
>> that probably does not change the space already allocated.)    Are there
> other
>> things we should be looking at?
>>> Our environment is FilePro 5.7.00D9 on a standalone Windows 7 PC
>>> with a
> 64-
>> bit OS and 16 GB of RAM.
>>> Thanks for any suggestions.
>>>
>>> Rich
>>>
>>> -------------- next part -------------- An HTML attachment was
>>> scrubbed...
>>> URL:
>>> <http://mailman.celestial.com/pipermail/filepro-list/attachments/201
>>> 80
>>> 702/d7549a78/attachment.html>
>>> _______________________________________________
>>> Filepro-list mailing list
>>> Filepro-list at lists.celestial.com
>>> Subscribe/Unsubscribe/Subscription Changes
>>> http://mailman.celestial.com/mailman/listinfo/filepro-list
>> _______________________________________________
>> Filepro-list mailing list
>> Filepro-list at lists.celestial.com
>> Subscribe/Unsubscribe/Subscription Changes
>> http://mailman.celestial.com/mailman/listinfo/filepro-list
> _______________________________________________
> Filepro-list mailing list
> Filepro-list at lists.celestial.com
> Subscribe/Unsubscribe/Subscription Changes
> http://mailman.celestial.com/mailman/listinfo/filepro-list
>

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus