dclerk Memory Corruption

Fairlight fairlite at fairlite.com
Sun Sep 6 12:13:32 PDT 2020 

Yes, yes I do.  As a matter of fact, I just recently updated LightMail,
OneGate, RawQuery, and OneBridge to all work with modern versions of the
Perl binaries and modules.  There are some deprecated (and now obsoleted)
calls in both a couple modules I use, as well as Perl itself.  At least
three of those are pre-2005 packages, so yes, over 15 years old.

According to you and Richard, I should charge people for the privilege of
obtaining the updated versions.  I'm not sure my clients would agree, but
I'll send them your way if they complain about it.  Thanks for convincing
me to charge upgrade fees, if I actually follow through with it.  :)

In the past, I have not charged for that courtesy.  LightMail has been
updated a few times at no cost to work with later Perl binaries.  This may
change, if you folks are justifying the behaviour of companies doing so.  I
can just tell the rest of the filePro community, "Hey, Richard Kreiss and
Nancy Palmquist said this was acceptable for software companies.  Complain
to them."  Considering 99% of those clients are -in- the filePro community,
you probably should have been more careful about that.  I now actually have
someone else to pin this on, if I decide I want to do it.  You're
respected figured in the community; you can both deal with it.  ;)

m->


On Sun, Sep 06, 2020 at 02:46:13PM -0400, Nancy Palmquist via Filepro-list thus spoke:
> If you are pinning this problem to an index, did you discuss reviewing the
> programming that is adding and deleting from that index to make sure it is
> sound.
> 
> And to Mark, does anyone support software that is over 15 years old to keep
> it compatible with new technologies and Operating Systems.  I have a shelf
> full of software that I paid good money to get and will not work on today's
> computers.  If you want to get versions of software that have been adjusted
> to accommodate more memory, new Operating systems, etc. You have to pay for
> the upgrades.  It is impossible to warranty software is going to work on
> computers and O/S systems that have not yet been released. fpTech does not
> have a crystal ball into what might be needed.  I am amazed that I have
> customers using new computers and working flawlessly with programming that
> was written in 1980.  That is 40 years ago.  I think that is big cudos for
> filePro.
> 
> Nancy
> 
> 
> On 9/4/2020 7:48 PM, Richard Kreiss via Filepro-list wrote:
> > I agree with Fairlight???s  recommendation to rebuild the index. However I suggest you delete the index in question from the command line and the check your disk for bad sectors and repair if necessary. Then rebuild the index. The corruption could be caused by a bad sector on the disk.
> > 
> > I have found , in the past, when a client has been having index issues, deleting it outside filePro offers a better chance of eliminating the  issue as dxmaint leave the header information. You want a ???clean slate??? when the rebuild takes place.
> > 
> > Richard Kreiss
> > GCC CONSULTING Consulting.
> > Sent from my iPhone
> > 
> > > On Sep 4, 2020, at 7:17 PM, Fairlight via Filepro-list <filepro-list at lists.celestial.com> wrote:
> > > 
> > > ???Your methodology looks like exactly what I use when I do it.  The
> > > difference is, rebuilding indexes stopped helping the last time I ran
> > > into it.
> > > 
> > > The index files are BTree+, if that helps you.  I know how the algorithm
> > > works.  There's an excellent visualiser for how it works.  How they
> > > implemented it on disk is a bit of a semi-opaque black box.  It's
> > > documented, but I always seem to have trouble going from algorithm -> file
> > > structure on disk.  At least so far, but I haven't take a bunch of time to
> > > nail it, because I don't really do things on spec anymore.  I may still
> > > do it in the future, but I have neither the time nor energy anytime soon.
> > > My Copious Free Time[tm] is better spent practising guitar and working on
> > > music for the foreseeable future.
> > > 
> > > m->
> > > 
> > > 
> > > On Fri, Sep 04, 2020 at 09:45:50PM +0000, Seijyaku thus spoke:
> > > > dclerk is being ran from a script, so I modified the script so that it would output trace files ??? see example below:
> > > > strace -e trace=all dclerk {clerk config} > /dev/null 2>> /tmp/strace.${$}
> > > > 
> > > > I let that run for 30 seconds and then collected my trace files to look for faults, which will appear as:
> > > > --- SIGABRT
> > > > --- SIGSEGV   <-- This is the segmentation fault I'm interested in
> > > > 
> > > > Just before the segmentation fault, there is a seek and read request on file descriptor 8: {These two lines do not change between failures}
> > > > lseek(8, 136192, SEEK_SET)              = 136192
> > > > read(8, ".\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
> > > > 
> > > > I tracked file descriptor 8 back to an open request for the assumed bad index ??? Do be careful as file descriptor numbers are reused, so make sure that you don't pass over a close call when backtracking.
> > > > open("/opt/filepro/lock/indexfile.C", O_RDWR) = 8
> > > > 
> > > > 
> > > > I am by no means an expert at this and welcome alternate views.  I have opened the index in a hex editor and reviewed the section it was reading directly before the segmentation fault but I don't see anything out of the ordinary ??? Without a full understanding of how the index file is laid out, I can only make assumptions.
> > > > 
> > > > In the end, all related errors have stopped after rebuilding this index.  I do have a backup of the faulty index in case the same index becomes corrupt again so that I can compare the two.
> > > > 
> > > > Thank you for all your help!
> > > > 
> > > > Sent with ProtonMail Secure Email.
> > > > 
> > > > ????????????????????? Original Message ?????????????????????
> > > > > On Friday, September 4, 2020 3:47 PM, Fairlight via Filepro-list <filepro-list at lists.celestial.com> wrote:
> > > > > 
> > > > > Rebuild the index in question with dxmaint.
> > > > > 
> > > > > I don't suppose you'd share your methodology for tracing them back to a
> > > > > specific index? ;)
> > > > > 
> > > > > m->
> > > > > 
> > > > > On Fri, Sep 04, 2020 at 08:29:37PM +0000, Seijyaku thus spoke:
> > > > > 
> > > > > > I have analyzed ten stack traces of clerk segmentation faults and tracked each of them back to a read on the same index file which is related to a lock table. This would seem to confirm your suspicion that the issue was related to a corrupted index.
> > > > > > What are my options for either validating or rebuilding the index?
> > > > > > Sent with ProtonMail Secure Email.
> > > > > > ????????????????????? Original Message ?????????????????????
> > > > > > On Thursday, September 3, 2020 4:52 PM, Fairlight via Filepro-list filepro-list at lists.celestial.com wrote:
> > > > > > 
> > > > > > > To me, it seems unpredictable, but not completely random. I've had
> > > > > > > bigger fish to fry, and this wasn't noticeably affecting customers, or
> > > > > > > it would have been a higher priority. I'd love it if our Apache logs
> > > > > > > would stop getting polluted, though.
> > > > > > > Good luck getting help from fP Tech, since they refuse to lift a finger
> > > > > > > to support 5.0.14 any longer, no matter how much you've already paid
> > > > > > > them. One of the 'lovely' things about that company. (Richard, save
> > > > > > > it. Nothing will sway me on this.)
> > > > > > > m->
> > > > > > > On Thu, Sep 03, 2020 at 09:33:00PM +0000, Seijyaku thus spoke:
> > > > > > > 
> > > > > > > > The server is running on VMware.
> > > > > > > > Do you know if this is limited to specific clerk calls or is it completely random? I'm still working to pin down what calls result in errors on my side.
> > > > > > > > Sent with ProtonMail Secure Email.
> > > > > > > > ????????????????????? Original Message ?????????????????????
> > > > > > > > On Thursday, September 3, 2020 3:48 PM, Fairlight via Filepro-list filepro-list at lists.celestial.com wrote:
> > > > > > > > 
> > > > > > > > > For what it's worth, I see this regularly with CentOS 7.8 (and when it
> > > > > > > > > was any 7.x starting around .4) on VMware 5.5.
> > > > > > > > > I originally thought it was some bad indexes, because it went away
> > > > > > > > > originally when we rebuilt a problem index. It does not actually seem
> > > > > > > > > to be that any longer.
> > > > > > > > > I literally do not know what causes it, but I would greatly benefit from
> > > > > > > > > hearing the explanation. The servers in question are never short on
> > > > > > > > > RAM when it's transpiring, and also only -some- of the hundreds of calls
> > > > > > > > > pre minute to clerk will do it, not all of them.
> > > > > > > > > It's not just you, though.
> > > > > > > > > You on VMware, or physical hardware? That would potentially rule out
> > > > > > > > > VMware if you're on physical hardware only.
> > > > > > > > > m->
> > > > > > > > > On Thu, Sep 03, 2020 at 07:04:26PM +0000, Seijyaku via Filepro-list thus spoke:
> > > > > > > > > 
> > > > > > > > > > CentOS 7.8.
> > > > > > > > > > Sent with ProtonMail Secure Email.
> > > > > > > > > > ????????????????????? Original Message ?????????????????????
> > > > > > > > > > On Thursday, September 3, 2020 2:02 PM, Ross Salas ross.salas at gmail.com wrote:
> > > > > > > > > > 
> > > > > > > > > > > Which operating system/version you running?
> > > > > > > > > > > On Thu, Sep 3, 2020 at 11:36 AM Seijyaku seijyaku+filepro at protonmail.com wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Below is the output from the Linux free command. I assume this is what you're asking for?
> > > > > > > > > > > > total used free shared buff/cache available
> > > > > > > > > > > > Mem: 32936704 2835928 2408340 1759048 27692436 27864508
> > > > > > > > > > > > Swap: 0 0 0
> > > > > > > > > > > > 32GB installed, about 3GB used, the majority as buffer/cache.
> > > > > > > > > > > > Sent with ProtonMail Secure Email.
> > > > > > > > > > > > ????????????????????? Original Message ?????????????????????
> > > > > > > > > > > > On Thursday, September 3, 2020 1:23 PM, Ross Salas via Filepro-list filepro-list at lists.celestial.com wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > what's the memory (RAM) status of that server?
> > > > > > > > > > > > > On Thu, Sep 3, 2020 at 10:58 AM Seijyaku via Filepro-list <
> > > > > > > > > > > > > filepro-list at lists.celestial.com> wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > I'm working on a server running dclerk version 5.0.14D4 that is
> > > > > > > > > > > > > > occasionally showing the error below in some of the logs. Any input on what
> > > > > > > > > > > > > > might be causing this or how to resolve it?
> > > > > > > > > > > > > > \x07,
> > > > > > > > > > > > > > *** Error in `dclerk': malloc(): memory corruption (fast): 0x08b0c2c8 ***,
> > > > > > > > > > > > > > ======= Backtrace: =========,
> > > > > > > > > > > > > > /lib/libc.so.6(+0x77dfc)[0xf7ddfdfc],
> > > > > > > > > > > > > > /lib/libc.so.6(+0x7ae98)[0xf7de2e98],
> > > > > > > > > > > > > > /lib/libc.so.6(__libc_malloc+0x9a)[0xf7de496a],
> > > > > > > > > > > > > > dclerk[0x80b1e4e],
> > > > > > > > > > > > > > dclerk[0x80b18e6],
> > > > > > > > > > > > > > dclerk[0x80b15f1],
> > > > > > > > > > > > > > dclerk[0x80b1e02],
> > > > > > > > > > > > > > dclerk[0x80ae0af],
> > > > > > > > > > > > > > dclerk[0x80af2f4],
> > > > > > > > > > > > > > dclerk[0x80aec39],
> > > > > > > > > > > > > > dclerk[0x80a95ad],
> > > > > > > > > > > > > > [0xf7f8bec0],
> > > > > > > > > > > > > > dclerk[0x80daa1d],
> > > > > > > > > > > > > > dclerk[0x80daf33],
> > > > > > > > > > > > > > dclerk[0x80d90d7],
> > > > > > > > > > > > > > dclerk[0x80d301b],
> > > > > > > > > > > > > > dclerk[0x80d66cd],
> > > > > > > > > > > > > > dclerk[0x80d61ef],
> > > > > > > > > > > > > > dclerk[0x80d6178],
> > > > > > > > > > > > > > dclerk[0x8070610],
> > > > > > > > > > > > > > dclerk[0x8061017],
> > > > > > > > > > > > > > dclerk[0x805fbb2],
> > > > > > > > > > > > > > dclerk[0x805e65f],
> > > > > > > > > > > > > > dclerk[0x805e907],
> > > > > > > > > > > > > > dclerk[0x8071bc2],
> > > > > > > > > > > > > > dclerk[0x804f2e7],
> > > > > > > > > > > > > > dclerk[0x804e230],
> > > > > > > > > > > > > > /lib/libc.so.6(__libc_start_main+0xf3)[0xf7d822a3],
> > > > > > > > > > > > > > ======= Memory map: ========,
> > > > > > > > > > > > > > 08048000-080f2000 r-xp 00000000 08:02 1614140597 /opt/filepro/dclerk,
> > > > > > > > > > > > > > 080f2000-08104000 rwxp 000aa000 08:02 1614140597 /opt/filepro/dclerk,
> > > > > > > > > > > > > > 08104000-08113000 rwxp 00000000 00:00 0 ,
> > > > > > > > > > > > > > 08aa0000-08b2a000 rwxp 00000000 00:00 0 [heap],
> > > > > > > > > > > > > > f7b00000-f7b21000 rwxp 00000000 00:00 0 ,
> > > > > > > > > > > > > > f7b21000-f7c00000 ---p 00000000 00:00 0 ,
> > > > > > > > > > > > > > f7cae000-f7cc7000 r-xp 00000000 08:02 1611440773
> > > > > > > > > > > > > > /usr/lib/libgcc_s-4.8.5-20150702.so.1,
> > > > > > > > > > > > > > f7cc7000-f7cc8000 r-xp 00018000 08:02 1611440773
> > > > > > > > > > > > > > /usr/lib/libgcc_s-4.8.5-20150702.so.1,
> > > > > > > > > > > > > > f7cc8000-f7cc9000 rwxp 00019000 08:02 1611440773
> > > > > > > > > > > > > > /usr/lib/libgcc_s-4.8.5-20150702.so.1,
> > > > > > > > > > > > > > f7cd5000-f7d43000 rwxp 00000000 00:00 0 ,
> > > > > > > > > > > > > > f7d43000-f7d53000 rwxs 00000000 00:01 2 /SYSVf102e0ed (deleted),
> > > > > > > > > > > > > > f7d53000-f7d5e000 r-xp 00000000 08:02 1611347286 /usr/lib/
> > > > > > > > > > > > > > libnss_files-2.17.so,
> > > > > > > > > > > > > > f7d5e000-f7d5f000 r-xp 0000a000 08:02 1611347286 /usr/lib/
> > > > > > > > > > > > > > libnss_files-2.17.so,
> > > > > > > > > > > > > > f7d5f000-f7d60000 rwxp 0000b000 08:02 1611347286 /usr/lib/
> > > > > > > > > > > > > > libnss_files-2.17.so,
> > > > > > > > > > > > > > f7d60000-f7d68000 rwxp 00000000 00:00 0 ,
> > > > > > > > > > > > > > f7d68000-f7f2c000 r-xp 00000000 08:02 1611317456 /usr/lib/libc-2.17.so,
> > > > > > > > > > > > > > f7f2c000-f7f2d000 ---p 001c4000 08:02 1611317456 /usr/lib/libc-2.17.so,
> > > > > > > > > > > > > > f7f2d000-f7f2f000 r-xp 001c4000 08:02 1611317456 /usr/lib/libc-2.17.so,
> > > > > > > > > > > > > > f7f2f000-f7f30000 rwxp 001c6000 08:02 1611317456 /usr/lib/libc-2.17.so,
> > > > > > > > > > > > > > f7f30000-f7f33000 rwxp 00000000 00:00 0 ,
> > > > > > > > > > > > > > f7f33000-f7f73000 r-xp 00000000 08:02 1611317464 /usr/lib/libm-2.17.so,
> > > > > > > > > > > > > > f7f73000-f7f74000 r-xp 0003f000 08:02 1611317464 /usr/lib/libm-2.17.so,
> > > > > > > > > > > > > > f7f74000-f7f75000 rwxp 00040000 08:02 1611317464 /usr/lib/libm-2.17.so,
> > > > > > > > > > > > > > f7f75000-f7f78000 r-xp 00000000 08:02 1610785783
> > > > > > > > > > > > > > /usr/lib/libtermcap.so.2.0.8,
> > > > > > > > > > > > > > f7f78000-f7f79000 r-xp 00002000 08:02 1610785783
> > > > > > > > > > > > > > /usr/lib/libtermcap.so.2.0.8,
> > > > > > > > > > > > > > f7f79000-f7f7a000 rwxp 00003000 08:02 1610785783
> > > > > > > > > > > > > > /usr/lib/libtermcap.so.2.0.8,
> > > > > > > > > > > > > > f7f85000-f7f87000 rwxp 00000000 00:00 0 ,
> > > > > > > > > > > > > > f7f87000-f7f8b000 r--p 00000000 00:00 0 [vvar],
> > > > > > > > > > > > > > f7f8b000-f7f8d000 r-xp 00000000 00:00 0 [vdso],
> > > > > > > > > > > > > > f7f8d000-f7faf000 r-xp 00000000 08:02 1610615685 /usr/lib/ld-2.17.so,
> > > > > > > > > > > > > > f7faf000-f7fb0000 r-xp 00021000 08:02 1610615685 /usr/lib/ld-2.17.so,
> > > > > > > > > > > > > > f7fb0000-f7fb1000 rwxp 00022000 08:02 1610615685 /usr/lib/ld-2.17.so,
> > > > > > > > > > > > > > ffa8e000-ffab0000 rwxp 00000000 00:00 0 [stack],
> > > > > > > > > > > > > > -------------- next part --------------
> > > > > > > > > > > > > > An HTML attachment was scrubbed...
> > > > > > > > > > > > > > URL: <
> > > > > > > > > > > > > > http://mailman.celestial.com/pipermail/filepro-list/attachments/20200828/4fed3c7e/attachment.html
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Filepro-list mailing list
> > > > > > > > > > > > > > Filepro-list at lists.celestial.com
> > > > > > > > > > > > > > Subscribe/Unsubscribe/Subscription Changes
> > > > > > > > > > > > > > http://mailman.celestial.com/mailman/listinfo/filepro-list
> > > > > > > > > > > > > -------------- next part --------------
> > > > > > > > > > > > > An HTML attachment was scrubbed...
> > > > > > > > > > > > > URL: http://mailman.celestial.com/pipermail/filepro-list/attachments/20200903/1b6e9be8/attachment.html
> > > > > > > > > > > > > Filepro-list mailing list
> > > > > > > > > > > > > Filepro-list at lists.celestial.com
> > > > > > > > > > > > > Subscribe/Unsubscribe/Subscription Changes
> > > > > > > > > > > > > http://mailman.celestial.com/mailman/listinfo/filepro-list
> > > > > > > > > > > > > -------------- next part --------------
> > > > > > > > > > > > > An HTML attachment was scrubbed...
> > > > > > > > > > > > > URL: http://mailman.celestial.com/pipermail/filepro-list/attachments/20200903/8727da64/attachment.html
> > > > > > > > > > Filepro-list mailing list
> > > > > > > > > > Filepro-list at lists.celestial.com
> > > > > > > > > > Subscribe/Unsubscribe/Subscription Changes
> > > > > > > > > > http://mailman.celestial.com/mailman/listinfo/filepro-list
> > > > > > > > > --
> > > > > > > > > Audio panton, cogito singularis.
> > > > > > > > > Filepro-list mailing list
> > > > > > > > > Filepro-list at lists.celestial.com
> > > > > > > > > Subscribe/Unsubscribe/Subscription Changes
> > > > > > > > > http://mailman.celestial.com/mailman/listinfo/filepro-list
> > > > > > > --
> > > > > > > Audio panton, cogito singularis.
> > > > > > > Filepro-list mailing list
> > > > > > > Filepro-list at lists.celestial.com
> > > > > > > Subscribe/Unsubscribe/Subscription Changes
> > > > > > > http://mailman.celestial.com/mailman/listinfo/filepro-list
> > > > > --
> > > > > 
> > > > > Audio panton, cogito singularis.
> > > > > 
> > > > > Filepro-list mailing list
> > > > > Filepro-list at lists.celestial.com
> > > > > Subscribe/Unsubscribe/Subscription Changes
> > > > > http://mailman.celestial.com/mailman/listinfo/filepro-list
> > > > 
> > > > 
> > > -- 
> > > Audio panton, cogito singularis.
> > > _______________________________________________
> > > Filepro-list mailing list
> > > Filepro-list at lists.celestial.com
> > > Subscribe/Unsubscribe/Subscription Changes
> > > http://mailman.celestial.com/mailman/listinfo/filepro-list
> > _______________________________________________
> > Filepro-list mailing list
> > Filepro-list at lists.celestial.com
> > Subscribe/Unsubscribe/Subscription Changes
> > http://mailman.celestial.com/mailman/listinfo/filepro-list
> 
> -- 
> Nancy Palmquist         MOS & filePro Training Available
> Virtual Software Systems    Web Based Training and Consulting
> PHONE: (412) 835-9417           Web site:  http://www.vss3.com
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mailman.celestial.com/pipermail/filepro-list/attachments/20200906/12ca7e94/attachment.html>
> _______________________________________________
> Filepro-list mailing list
> Filepro-list at lists.celestial.com
> Subscribe/Unsubscribe/Subscription Changes
> http://mailman.celestial.com/mailman/listinfo/filepro-list
> 

-- 
Audio panton, cogito singularis.

More information about the Filepro-list mailing list