fP and data integrity...how to maintain on non-quiescent systems?

[GCC Consulting]

> -----Original Message-----
> From: 
> filepro-list-bounces+gccconsulting=comcast.net at lists.celestial
> .com 
> [mailto:filepro-list-bounces+gccconsulting=comcast.net at lists.c
> elestial.com] On Behalf Of Fairlight
> Sent: Saturday, February 17, 2007 10:57 PM
> To: filepro-list at lists.celestial.com
> Subject: Re: fP and data integrity...how to maintain on 
> non-quiescent systems?
> 
> On Sat, Feb 17, 2007 at 07:11:20PM -0800, Bill Campbell may 
> or may not have proven themselves an utter git by pronouncing:
> > You might look at the way that MySQL and openldap handle database 
> > replication.  They keep a log file of all transactions updating the 
> > database(s), which is the used by the secondary machine(s) 
> to update 
> > their copies of the data.
> 
> Yeah, I know how MySQL does it.  Although they -just- got a 
> break from the master/slave replication and got into 
> omni-directional replication in 5.x from what I remember 
> reading.  (I've always wondered what happens if the remote 
> machine is down, but I'm going to assume it was sanely enough 
> written to be fault tolerant.)
> 
> > This is a bit more complex that simple rsyncs, but it has the 
> > advantage that it can work with live systems, and when 
> doing backups, 
> > it requires almost zero time to switch servers.
> 
> Oh, it's the way to go, no doubt.  I was, personally, trying 
> explicitly
> -not- to bring MySQL into it, from the standpoint of someone 
> or other piping in with, "Well if MySQL is so great, go use 
> it--this is filePro."
> So I'm purely looking at it from the standpoint of what fP 
> has and doesn't have that could facilitate this, without 
> trying to draw comparison.
> Drawing the comparison won't really solve the problem 
> anyway--even though the design difference is obviously the 
> answer on one side of the fence.
> 
> > Given that FilePro doesn't have this type of logging, it 
> would require 
> > writing your own logging routines to generate the 
> transaction logs.  
> > As a first cut, I would probably write out the appropriate SQL 
> > commands to perform the operations.  One might even be able to hack 
> > SQLlite to work with FilePro's binary data formats.
> 
> Yeah, I was kinda hoping that it was something that would 
> involve less than a month or so to develop and test.  :)  
> Someone is hot to have all their data synced over at 
> (*chuckle*) 15min intervals max.  Given the way the data is 
> laid out (it's not just fP, it's three trees, only one of 
> which is filepro's PFDIR) and the fact that we're dealing 
> with 1GB key files in some places in filePro, and in the 
> other rsync we have a tree that literally took 15min to build 
> the file list when it needed minimal changes...  It's just not good.
> 
> Transactional replication would be idea., but fP doesn't even 
> have a concept of "commit"...unless it's tennable to open and 
> make all lookups, do no explicit closes, and just let them 
> auto-close when you hit END and change records...they'd all 
> auto-close in pretty close succession, but not guaranteed 
> against race conditions.  But even if it had a commit, you're 
> still looking at having to basically rewrite transactions and 
> replication both from the ground up.  I started a discussion 
> about that about half a year ago and when it got down to the 
> system maintained fields, timezone offset deltas, etc., it 
> got much deeper than I'd want to go--especially writing it in 
> a language I'm not 100% comfy in for daily use.
> 
> Now, you and I both know what one optimal solution is.  And I 
> agree with you.  But I'm still wondering what the people out 
> there that have done filePro and almost only filePro for 25+ 
> years have done to eliminate this problem all this time.  
> Still coming up blank.  What, does everyone just ignore the 
> problem, pray for the best, and restore what they can, best 
> they can when things go south?  It's the weekend, so I'll 
> give it a few days.  I just have this nagging suspicion there 
> won't be a "real" answer due to fP's architecture--and that 
> the problem -is- largely ignored.  I look forward to being 
> proved wrong, really.

How about using mirrored servers and then backing up the mirror at some
point?

Not the best solution but you would have all of the data in sync and would
be able to switch over to the other server fairly rapidly.

There is software out there, at least on the wiondows side, which allow easy
mirroring of servers.

Richard Kreiss
GCC Consulting