fP and data integrity...how to maintain on non-quiescent systems?

[Fairlight]

On Sat, Feb 17, 2007 at 07:11:20PM -0800, Bill Campbell may or may not have
proven themselves an utter git by pronouncing:
> You might look at the way that MySQL and openldap handle database
> replication.  They keep a log file of all transactions updating the
> database(s), which is the used by the secondary machine(s) to update their
> copies of the data.

Yeah, I know how MySQL does it.  Although they -just- got a break from the
master/slave replication and got into omni-directional replication in 5.x
from what I remember reading.  (I've always wondered what happens if the
remote machine is down, but I'm going to assume it was sanely enough
written to be fault tolerant.)

> This is a bit more complex that simple rsyncs, but it has the advantage
> that it can work with live systems, and when doing backups, it requires
> almost zero time to switch servers.

Oh, it's the way to go, no doubt.  I was, personally, trying explicitly
-not- to bring MySQL into it, from the standpoint of someone or other
piping in with, "Well if MySQL is so great, go use it--this is filePro."
So I'm purely looking at it from the standpoint of what fP has and doesn't
have that could facilitate this, without trying to draw comparison.
Drawing the comparison won't really solve the problem anyway--even though
the design difference is obviously the answer on one side of the fence.

> Given that FilePro doesn't have this type of logging, it would require
> writing your own logging routines to generate the transaction logs.  As a
> first cut, I would probably write out the appropriate SQL commands to
> perform the operations.  One might even be able to hack SQLlite to work
> with FilePro's binary data formats.

Yeah, I was kinda hoping that it was something that would involve less than
a month or so to develop and test.  :)  Someone is hot to have all their
data synced over at (*chuckle*) 15min intervals max.  Given the way the
data is laid out (it's not just fP, it's three trees, only one of which is
filepro's PFDIR) and the fact that we're dealing with 1GB key files in some
places in filePro, and in the other rsync we have a tree that literally
took 15min to build the file list when it needed minimal changes...  It's
just not good.

Transactional replication would be idea., but fP doesn't even have a
concept of "commit"...unless it's tennable to open and make all lookups, do
no explicit closes, and just let them auto-close when you hit END and
change records...they'd all auto-close in pretty close succession, but not
guaranteed against race conditions.  But even if it had a commit, you're
still looking at having to basically rewrite transactions and replication
both from the ground up.  I started a discussion about that about half a
year ago and when it got down to the system maintained fields, timezone
offset deltas, etc., it got much deeper than I'd want to go--especially
writing it in a language I'm not 100% comfy in for daily use.

Now, you and I both know what one optimal solution is.  And I agree with
you.  But I'm still wondering what the people out there that have done
filePro and almost only filePro for 25+ years have done to eliminate this
problem all this time.  Still coming up blank.  What, does everyone just
ignore the problem, pray for the best, and restore what they can, best they
can when things go south?  It's the weekend, so I'll give it a few days.  I
just have this nagging suspicion there won't be a "real" answer due to fP's
architecture--and that the problem -is- largely ignored.  I look forward to
being proved wrong, really.

mark->