fP transaction-based clustering - viability?

[Brian K. White]

Fairlight wrote:
> Hi All,
>
> I'm curious how many people have multiple fP servers out there which
> have data on them that they wish to keep synchronised in
> near-realtime (say less than 2min propogation) or in -actual- full
> realtime, in a transactional fashion.
>
> Several people are using rsync, ftp, scp, et al. as more periodic
> (nightly or a few times per day) backup/sync "solutions", but that
> simply doesn't cut it if you have two or five web servers, all of
> which are supposed to be load-balanced and serving the same data at
> any given point, or if you have multiple servers which are used to
> balance the load of internal operations but run into the same
> issue--all data isn't always present on every server immediately.
> And even rsync isn't safe for realtime work when files are in use.
> Too many timing issues.  NFS has its definite downsides--even using
> v3 of NFS.

Just a little idea about the rsync...
If you rsync often enough, and/or optimize the job enough that it just 
includes the necessary files which might be a lot less than everything in a 
filepro tree, then the job can go fast enough that it's OK for the web 
service to be unresponsive for that time.

You could have the various servers agree upon a time to rsync and/or have a 
master server throw lockfiles up on the others or send a special cgi request 
that tells the others to finish up what they're doing and que further cgi 
requests but don't access fp any more. And wait for corresponding "ok I'm 
done with my pending tasks" files (lock-acknowledge-files)?

Once all the other servers have acknowledges the lock, run the rsyncs, 
release the locks.

That will keep the data uncorrupted, a little old at worst few minutes to 
few hours, and from the users point of view the worst case is they hit 
"submit" just at the beginning of a lock-down and the cgi says "working..." 
for a few seconds before returning the answer instead of returning it 
instantly. (the cgi would queu-up the job and process it as soon as the lock 
is released)

And even then it only works as long as the sub-servers are just serving up 
data sort of read-only. If the sub servers need to make data, then I don't 
even know of a way for rsync to do it unless you do basically the same 
programming or even more, as it would take to do transactions anyways.

This is not a real solution of course, you are absolutely correct in that we 
need transactions. I have toyed with a simple cgi script a couple years ago 
that is basically just a wrapper for rreport + export word + selection 
processing that parsed the query-string.  You submit a simple selector 
sentence as the query-string cgi (including use of @td etc...) and the cgi 
spits out csv records of the filepro records it found.

The idea was to develop it further bit at a time along these lines:
to-do:
* expand selector sentence to be a wrapper for a full selection table. 
ideally by just writing a temp sel file using it and removing it, maybe in a 
ramdisk or maybe normal disk cache will keep it from ever hitting the disk 
anyways.
* generalize the code so that instead of working out of a particular data 
file, it works out of a central standing file and can access any desired 
data file.
* further generalize so that the cgi can request fields out of different 
files to assemble a record that doesn't correspond directly to any filepro 
file record, and do selection processing on those fiedls at will too.
* allow posting data as well as just reading.
* secure it all somehow

Once you could post, you'd have a basic transaction mechanism. One filepro 
process could talk to anothers cgi script via system + wget or curl, or user 
and netcat, or maybe it could just be done directly netcat to netcat. Using 
the web server and a cgi script just makes it easy to go over the public net 
without having to set up special nat forwarding rules and firewall 
exceptions all the time so you could transact between distant machines and 
different companies.

Then I figured I might be reinventing sql or something and in any event the 
customer who wanted it stopped wanting it and so it didn't get any further 
than the read-only, one-file, + map of future development stage.

Brian K. White  --  brian at aljex.com  --  http://www.aljex.com/bkw/
+++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
filePro BBx  Linux SCO  Prosper/FACTS AutoCAD  #callahans Satriani