Indexes Gone Wild

[Bruce Easton]

Barry Wiseman wrote Monday, May 19, 2008 10:12 PM:
> SunOS 5.9
> filePro 5.0.13
> 
> I'm working on why indexes at this site keep getting corrupted (don't 
> reliably show records which are in fact in the file; index rebuild 
> helps), watching carefully in the evening when there are only several 
> users active instead of hundreds.
> 
> I'm observing the following curious phenomenon with a known bad index.A. 
>    The mod timestamp on this index is typically newer than the key or 
> other indexes, sometimes by a lot.  Here's a snapshot showing a 
> seven-minute lag:
> 
> #pwd
> /u/appl/filepro/notelns
> Trinity #l -tr key index.*
> -rw-------   1 filepro  other    345012318 May 19 21:46 key
> -rw-------   1 filepro  other    537054208 May 19 21:46 index.P
> -rw-------   1 filepro  other    524056576 May 19 21:46 index.O
> -rw-------   1 filepro  other    482894848 May 19 21:46 index.F
> -rw-------   1 filepro  other    422981632 May 19 21:46 index.E
> -rw-------   1 filepro  other    294321152 May 19 21:46 index.D
> -rw-------   1 filepro  other    178771968 May 19 21:46 index.C
> -rw-------   1 filepro  other    57499997 May 19 21:46 index.B
> -rw-------   1 filepro  other    122174464 May 19 21:53 index.A
> 
> Eventually, key and all indexes are updated in the expected manner, like 
> this:
> 
> #l -tr key index.*
> -rw-------   1 filepro  other    345012570 May 19 22:07 key
> -rw-------   1 filepro  other    537054208 May 19 22:07 index.P
> -rw-------   1 filepro  other    524056576 May 19 22:07 index.O
> -rw-------   1 filepro  other    482894848 May 19 22:07 index.F
> -rw-------   1 filepro  other    422981632 May 19 22:07 index.E
> -rw-------   1 filepro  other    294321152 May 19 22:07 index.D
> -rw-------   1 filepro  other    178772992 May 19 22:07 index.C
> -rw-------   1 filepro  other    57500039 May 19 22:07 index.B
> -rw-------   1 filepro  other    122174464 May 19 22:07 index.A
> 
> Then some dang program starts writing index.A only all over again.
> 
> This file is maintained only via lookups from other files.  The index is 
> built on four fields: (8,.0),(14,*),(8,mdy/),(3,.0).  At moments like 
> the above when index.A but nothing else has been written, there are no 
> record locks on the key or any of the indexes.  (No, there aren't any 
> keyx* segments.)
> 
> What sort of filePro code could cause clerk to write index.A without 
> writing the key?  Or has this OS simply lost its marbles?
> 

Is it possible that the program has been corrupting one or more 
indexes all along, and that there was some previous (prior to your 
involvement) program run to rebuild index A? (Something initiated 
by cron comes to mind first, but I guess someone could have 
hidden a system command for dxmaint inside a filepro program and 
that maybe gets kicked off when some other program [i.e., nightly 
totalling program] gets run.)  Have you grep'd for dxmaint in 
processing and checked the cron?

Bruce

Bruce Easton
STN, Inc.