fP and data integrity...how to maintain on non-quiescent systems?

[John Esak]

IF the systems are truly mirrored... how and when would you do a backup of
the mirror... one that wouldn't temporarily break the mirror that is...?

John


> -----Original Message-----
> From: filepro-list-bounces+john=valar.com at lists.celestial.com
> [mailto:filepro-list-bounces+john=valar.com at lists.celestial.com]On
> Behalf Of GCC Consulting
> Sent: Sunday, February 18, 2007 4:14 PM
> To: 'Fairlight'; filepro-list at lists.celestial.com
> Subject: RE: fP and data integrity...how to maintain on non-quiescent
> systems?
>
>
>
>
> > -----Original Message-----
> > From:
> > filepro-list-bounces+gccconsulting=comcast.net at lists.celestial
> > .com
> > [mailto:filepro-list-bounces+gccconsulting=comcast.net at lists.c
> > elestial.com] On Behalf Of Fairlight
> > Sent: Saturday, February 17, 2007 10:57 PM
> > To: filepro-list at lists.celestial.com
> > Subject: Re: fP and data integrity...how to maintain on
> > non-quiescent systems?
> >
> > On Sat, Feb 17, 2007 at 07:11:20PM -0800, Bill Campbell may
> > or may not have proven themselves an utter git by pronouncing:
> > > You might look at the way that MySQL and openldap handle database
> > > replication.  They keep a log file of all transactions updating the
> > > database(s), which is the used by the secondary machine(s)
> > to update
> > > their copies of the data.
> >
> > Yeah, I know how MySQL does it.  Although they -just- got a
> > break from the master/slave replication and got into
> > omni-directional replication in 5.x from what I remember
> > reading.  (I've always wondered what happens if the remote
> > machine is down, but I'm going to assume it was sanely enough
> > written to be fault tolerant.)
> >
> > > This is a bit more complex that simple rsyncs, but it has the
> > > advantage that it can work with live systems, and when
> > doing backups,
> > > it requires almost zero time to switch servers.
> >
> > Oh, it's the way to go, no doubt.  I was, personally, trying
> > explicitly
> > -not- to bring MySQL into it, from the standpoint of someone
> > or other piping in with, "Well if MySQL is so great, go use
> > it--this is filePro."
> > So I'm purely looking at it from the standpoint of what fP
> > has and doesn't have that could facilitate this, without
> > trying to draw comparison.
> > Drawing the comparison won't really solve the problem
> > anyway--even though the design difference is obviously the
> > answer on one side of the fence.
> >
> > > Given that FilePro doesn't have this type of logging, it
> > would require
> > > writing your own logging routines to generate the
> > transaction logs.
> > > As a first cut, I would probably write out the appropriate SQL
> > > commands to perform the operations.  One might even be able to hack
> > > SQLlite to work with FilePro's binary data formats.
> >
> > Yeah, I was kinda hoping that it was something that would
> > involve less than a month or so to develop and test.  :)
> > Someone is hot to have all their data synced over at
> > (*chuckle*) 15min intervals max.  Given the way the data is
> > laid out (it's not just fP, it's three trees, only one of
> > which is filepro's PFDIR) and the fact that we're dealing
> > with 1GB key files in some places in filePro, and in the
> > other rsync we have a tree that literally took 15min to build
> > the file list when it needed minimal changes...  It's just not good.
> >
> > Transactional replication would be idea., but fP doesn't even
> > have a concept of "commit"...unless it's tennable to open and
> > make all lookups, do no explicit closes, and just let them
> > auto-close when you hit END and change records...they'd all
> > auto-close in pretty close succession, but not guaranteed
> > against race conditions.  But even if it had a commit, you're
> > still looking at having to basically rewrite transactions and
> > replication both from the ground up.  I started a discussion
> > about that about half a year ago and when it got down to the
> > system maintained fields, timezone offset deltas, etc., it
> > got much deeper than I'd want to go--especially writing it in
> > a language I'm not 100% comfy in for daily use.
> >
> > Now, you and I both know what one optimal solution is.  And I
> > agree with you.  But I'm still wondering what the people out
> > there that have done filePro and almost only filePro for 25+
> > years have done to eliminate this problem all this time.
> > Still coming up blank.  What, does everyone just ignore the
> > problem, pray for the best, and restore what they can, best
> > they can when things go south?  It's the weekend, so I'll
> > give it a few days.  I just have this nagging suspicion there
> > won't be a "real" answer due to fP's architecture--and that
> > the problem -is- largely ignored.  I look forward to being
> > proved wrong, really.
>
> How about using mirrored servers and then backing up the mirror at some
> point?
>
> Not the best solution but you would have all of the data in sync and would
> be able to switch over to the other server fairly rapidly.
>
> There is software out there, at least on the wiondows side, which
> allow easy
> mirroring of servers.
>
> Richard Kreiss
> GCC Consulting
>
>
>
> _______________________________________________
> Filepro-list mailing list
> Filepro-list at lists.celestial.com
> http://mailman.celestial.com/mailman/listinfo/filepro-list