Indexes Gone Wild

[Bruce Easton]

Barry Wiseman wrote Tuesday, May 20, 2008 11:22 AM:
> Bruce Easton wrote:
> > Barry Wiseman wrote Monday, May 19, 2008 10:12 PM:
> >> SunOS 5.9
> >> filePro 5.0.13
> >>
> >> I'm working on why indexes at this site keep getting corrupted (don't
> >> reliably show records which are in fact in the file; index rebuild
> >> helps), watching carefully in the evening when there are only several
> >> users active instead of hundreds.
> >>
> >> I'm observing the following curious phenomenon with a known
> bad index.A.
> >>    The mod timestamp on this index is typically newer than the key or
> >> other indexes, sometimes by a lot.  Here's a snapshot showing a
> >> seven-minute lag:
> >>
> >> #pwd
> >> /u/appl/filepro/notelns
> >> Trinity #l -tr key index.*
> >> -rw-------   1 filepro  other    345012318 May 19 21:46 key
> >> -rw-------   1 filepro  other    537054208 May 19 21:46 index.P
> >> -rw-------   1 filepro  other    524056576 May 19 21:46 index.O
> >> -rw-------   1 filepro  other    482894848 May 19 21:46 index.F
> >> -rw-------   1 filepro  other    422981632 May 19 21:46 index.E
> >> -rw-------   1 filepro  other    294321152 May 19 21:46 index.D
> >> -rw-------   1 filepro  other    178771968 May 19 21:46 index.C
> >> -rw-------   1 filepro  other    57499997 May 19 21:46 index.B
> >> -rw-------   1 filepro  other    122174464 May 19 21:53 index.A
> >>
> >> Eventually, key and all indexes are updated in the expected
> manner, like
> >> this:
> >>
> >> #l -tr key index.*
> >> -rw-------   1 filepro  other    345012570 May 19 22:07 key
> >> -rw-------   1 filepro  other    537054208 May 19 22:07 index.P
> >> -rw-------   1 filepro  other    524056576 May 19 22:07 index.O
> >> -rw-------   1 filepro  other    482894848 May 19 22:07 index.F
> >> -rw-------   1 filepro  other    422981632 May 19 22:07 index.E
> >> -rw-------   1 filepro  other    294321152 May 19 22:07 index.D
> >> -rw-------   1 filepro  other    178772992 May 19 22:07 index.C
> >> -rw-------   1 filepro  other    57500039 May 19 22:07 index.B
> >> -rw-------   1 filepro  other    122174464 May 19 22:07 index.A
> >>
> >> Then some dang program starts writing index.A only all over again.
> >>
> >> This file is maintained only via lookups from other files.
> The index is
> >> built on four fields: (8,.0),(14,*),(8,mdy/),(3,.0).  At moments like
> >> the above when index.A but nothing else has been written, there are no
> >> record locks on the key or any of the indexes.  (No, there aren't any
> >> keyx* segments.)
> >>
> >> What sort of filePro code could cause clerk to write index.A without
> >> writing the key?  Or has this OS simply lost its marbles?
> >>
> >
> > Is it possible that the program has been corrupting one or more
> > indexes all along, and that there was some previous (prior to your
> > involvement) program run to rebuild index A? (Something initiated
> > by cron comes to mind first, but I guess someone could have
> > hidden a system command for dxmaint inside a filepro program and
> > that maybe gets kicked off when some other program [i.e., nightly
> > totalling program] gets run.)  Have you grep'd for dxmaint in
> > processing and checked the cron?
> >
> > Bruce
> >
> > Bruce Easton
> > STN, Inc.
> >
>
> Good ideas, Bruce.  There are some system "dxmaint" commands buried in
> processing, they seem to mostly relate to demand indexes and archival
> qualifiers, I will go carefully through all that.  They used to have
> many cron scripts rebuilding indexes nightly and weekly, the problem
> being that users are potentially active 24/7.  At my recommendation all
> that was unplugged a month ago, following which we scheduled 24 hours of
> downtime and rebuild every index in the system.
>
> I think the "writes" to index.A I thought I was seeing are a red
> herring, though; I see that, although the mod time of the file advances,
> the checksum is not changing as a result.

With regard to what I suggested about grep'ing for dxmaint - I blurted
that out like it works or not, but I have seen unexpected results on at
least a couple of older Sun systems with grep/fgrep (or where one of
those commands just didn't work at all).

Of more concern might be if there are prc tables there that could
contain dxmaint that are not saved in ascii.

Bruce

Bruce Easton
STN, Inc.