Indexes Gone Wild

[Barry Wiseman]

Bruce Easton wrote:
> Barry Wiseman wrote Monday, May 19, 2008 10:12 PM:
>> SunOS 5.9
>> filePro 5.0.13
>>
>> I'm working on why indexes at this site keep getting corrupted (don't 
>> reliably show records which are in fact in the file; index rebuild 
>> helps), watching carefully in the evening when there are only several 
>> users active instead of hundreds.
>>
>> I'm observing the following curious phenomenon with a known bad index.A. 
>>    The mod timestamp on this index is typically newer than the key or 
>> other indexes, sometimes by a lot.  Here's a snapshot showing a 
>> seven-minute lag:
>>
>> #pwd
>> /u/appl/filepro/notelns
>> Trinity #l -tr key index.*
>> -rw-------   1 filepro  other    345012318 May 19 21:46 key
>> -rw-------   1 filepro  other    537054208 May 19 21:46 index.P
>> -rw-------   1 filepro  other    524056576 May 19 21:46 index.O
>> -rw-------   1 filepro  other    482894848 May 19 21:46 index.F
>> -rw-------   1 filepro  other    422981632 May 19 21:46 index.E
>> -rw-------   1 filepro  other    294321152 May 19 21:46 index.D
>> -rw-------   1 filepro  other    178771968 May 19 21:46 index.C
>> -rw-------   1 filepro  other    57499997 May 19 21:46 index.B
>> -rw-------   1 filepro  other    122174464 May 19 21:53 index.A
>>
>> Eventually, key and all indexes are updated in the expected manner, like 
>> this:
>>
>> #l -tr key index.*
>> -rw-------   1 filepro  other    345012570 May 19 22:07 key
>> -rw-------   1 filepro  other    537054208 May 19 22:07 index.P
>> -rw-------   1 filepro  other    524056576 May 19 22:07 index.O
>> -rw-------   1 filepro  other    482894848 May 19 22:07 index.F
>> -rw-------   1 filepro  other    422981632 May 19 22:07 index.E
>> -rw-------   1 filepro  other    294321152 May 19 22:07 index.D
>> -rw-------   1 filepro  other    178772992 May 19 22:07 index.C
>> -rw-------   1 filepro  other    57500039 May 19 22:07 index.B
>> -rw-------   1 filepro  other    122174464 May 19 22:07 index.A
>>
>> Then some dang program starts writing index.A only all over again.
>>
>> This file is maintained only via lookups from other files.  The index is 
>> built on four fields: (8,.0),(14,*),(8,mdy/),(3,.0).  At moments like 
>> the above when index.A but nothing else has been written, there are no 
>> record locks on the key or any of the indexes.  (No, there aren't any 
>> keyx* segments.)
>>
>> What sort of filePro code could cause clerk to write index.A without 
>> writing the key?  Or has this OS simply lost its marbles?
>>
> 
> Is it possible that the program has been corrupting one or more 
> indexes all along, and that there was some previous (prior to your 
> involvement) program run to rebuild index A? (Something initiated 
> by cron comes to mind first, but I guess someone could have 
> hidden a system command for dxmaint inside a filepro program and 
> that maybe gets kicked off when some other program [i.e., nightly 
> totalling program] gets run.)  Have you grep'd for dxmaint in 
> processing and checked the cron?
> 
> Bruce
> 
> Bruce Easton
> STN, Inc.
>   

Good ideas, Bruce.  There are some system "dxmaint" commands buried in 
processing, they seem to mostly relate to demand indexes and archival 
qualifiers, I will go carefully through all that.  They used to have 
many cron scripts rebuilding indexes nightly and weekly, the problem 
being that users are potentially active 24/7.  At my recommendation all 
that was unplugged a month ago, following which we scheduled 24 hours of 
downtime and rebuild every index in the system.

I think the "writes" to index.A I thought I was seeing are a red 
herring, though; I see that, although the mod time of the file advances, 
the checksum is not changing as a result.