ADV: fP and changing the web transaction paradigm.

[Fairlight]

Since 1994, we have all been doing web-based transactions in pretty much
the same way when doing them from filePro:

1) Write a data file (depending on the client).
2) Execute an HTTP[S] client and process a single transaction.
3) Read our data back.

It's a stable paradigm, and a time-honoured and proven methodology for
achieving many goals.  Solid, but not impressive.

What happens, however, when response times escalate to the point of 3-7
seconds, and you have thousands of transactions to handle in short order?
In serial, this takes relative ages.  It's inefficient for large numbers of
transactions which could otherwise be batched to a single endpoint.  The
longer the individual transaction times, the longer sequential processing
takes, leading to potential bottlenecks and backlogs.

I have now back-ported threading technology into my SuperQuery client.
SuperQuery was never actually formally released publicly, although earlier,
non-threading versions are in use by multiple businesses.  SuperQuery
itself is an advanced fork of my RawQuery client (which actually has far
more users), which also handles SOAP and REST, in addition to normal CGI.
RawQuery only handled pure CGI encoding.  SuperQuery has seen production
use for about six years at various locations.  I've never had a single
bug report (which is more than I can say for certain other clients and
libraries).

With the new threading code in play, plus a brand new batch mode, if
you need to conduct hundreds or thousands of transactions to the same
URL endpoint in one go, you can save up to 80-97% time on the entire
batch transaction time, as you do the web requests in parallel, rather
than sequentially.  The amount of time saved varies depending on how
many concurrent threads your endpoint will allow you to use.  You need
only five concurrent connections to achieve ~80% gains, assuming a
two-second transaction time on average.  It gets better the more concurrent
connections you have, although diminishing returns kick in at the point
where thread management time starts to outstrip transaction times.

The new paradigm is:

1) Write -all- your query data files (with numeric index prefixes).
2) Execute the web client program once in batch mode.
3) Read -all- your query response files (correlating the prefixes).

Additionally, SuperQuery, in its latest incarnation with threading,
possesses a byte-range mode for HTTP[S], allowing you to download a file
which should take ~10 minutes in about a third of that time.  It's an
implementation of segmented downloading, and is great for fetching large
files like ISO images.  Obviously the time savings in segmented downloads
depends upon your available bandwidth, as with any web client.

The batch mode is impressive, in terms of performance gains.  What takes
about 1hr36min for 2500 serial 2s-long transactions can be done in
9min20sec or so over the same transaction count and transaction lengths.
It's not only faster, it's actually easier on system CPU, as it doesn't
need to repeatedly allocate memory and reload the same shared libraries and
do the same setup steps over thousands of executions.  Those steps all take
system CPU resources, as well as extra time; you can actually gain about
9% performance speed just by running the batch mode without threading, in
serial yet.  (Batch mode can be run in serial or parallel.)

I'm not putting this out broadly as a completely canned package with one
uniform price.  This functionality has been back-ported into SuperQuery
for enterprise-grade use.  We're talking 24/7/365 enterprise-grade use,
where it's being used day in, day out to make a profit at the heart of your
enterprise.  If you have need of such a creature, you can contact me to
discuss your use case, and we'll explore what it would cost to license for
your purposes.

The new SuperQuery is available for both *nix and Windows.  It still
handles single transactions, but the really amazing value is in the
new threaded batch mode and the segmented byte-range downloading.
It's perfectly capable of being a one-stop-shoppe for B2B web client
transactions, though.

If you're interested, feel free to reach out in private.

Everyone have a Merry Christmas (or other holiday of your choosing)!

Bests,

mark->
-- 
Audio panton, cogito singularis.