xfer problems sco-sun

[Bill Vermillion]

On Sat, Apr 03, 2004 at 11:38:03AM -0500, Jay R. Ashworth thus spoke:
> On Sat, Apr 03, 2004 at 08:57:52AM -0500, Bill Vermillion wrote:
> > > Ending the files with a line feed would make it easier to
> > > manipulate them with standard tools.

> > What tools can be defined as standard.

> "The tools that ship with my distribution."

Oh, those tools :-)

> > I've seen many thing over the years with built in line length
> > limits.  One problem I recall with at least earlier MS
> > implementations was line length limitations in the order of 500
> > bytes.  I don't know if that has been changed.

> Excuse me: "The tools that ship with my *Linux* distribution."

The past few years have seen a lot of the old limits being changed
- to higher limits usually.  Too often the limits are worked around
by changing existing things, while in reality they should undergo a
complete redesign.  Of course items that were limited because of
definitions of such as integers often work with just a redefinition
of that, but cause problems interacting with others that have not
changed.  An example was the checking for free space on a file
system based on a 2GB file size limit in FP, which required an
environmental variable that instructs the program to ignore those
limits.

It doesn't seem that long ago that the MS OSes were limited to
500MB drives, and the originals were only 30MB with the 12 bit FAT
as I recall.  Then there was the requriement to boot withing the
first 1024 cylinders, and that was worked around by using LBA
to fake things out.  At least the limits are getting bigger - and I
don't know that I'll live to see the day where I can't manage to
fit the boot information into the first 1.5 Terabytes of
filesystem that is the limit on my system.  It will probably be
awhile [if ever] that I get to that size.

But upper limits are constantly being raised as evidenced by
SGI's CXFS file system that they designed for the SAN environment.
It's called "The InfiniteStorage Shared File System".

While nothing is infinite, for all practical purposes this is.
You have a file SIZE limit of 9 million Terabytes [ or 9000
Petabytes or 9 Exabytes, if I have my prefixes straight].
The file SYSTEM limit is 18 million TB so you can only store
two 9 million TB files on it :-)

It is designed to be transparent file storage across platforms as
Unix, Linux, MS, and doesn't worry about content.  Media content is
huge and requires fast movement.  That's why some of the RAID
systems are made of many smaller drives to get the sustained
bandwidth high enough and run on FC to get 2Gb/sec transfer rates.

> > I remember on old Unix systems when there were few tools going into
> > a hex editor to add 'nl's ever so often in a large file in order to
> > get it to read into vi with the the 'line too long' error.

> Yeah; vim doesn't have that problem; I've successfully read and
> written multi-megabyte files with 3 (count 'em, 3) LF's.

I've never like 'vim' and and happy with 'nvi' - the natural
evolution of Joy's original vi.  The one thing I've noticed in
Linux distributions is that the defaults seem to change with
distributions and I've hit more than one that were impossible to
run without changing because of my inability to easily detect red
from a black background because the the low luminance value.  

...

> > My personal opinion of the MS line terminated approach to
> > fixed-length databases, is that they didn't seem to grasp the idea
> > of fixed length, as so many in the MS world early on seem to come
> > from a sequential access world, and the speed, or lack thereof,
> > showed this approach.
> > 
> > However many databases are still stuck in the past and I've seen at
> > least a couple that take advantage of more modern approaches and
> > the standard editing tools would never work.

> Indeed.  But, as the other poster noted, "fixed length" is not
> incompatible with "looks like a text file".  As long as you're careful
> with your text editor, it can be a very useful technique in certain
> circumstances.

But fixed length has no definiton on the bytes/bits contained in
the defined length of that record.  What gets interesting is when
you run across something that stores dates in packed form based on
perhaps an arbitrary start date decreed by the developer, or based
on the first year of release.  A glance at my news history file
shows that somewhere in the past hour it was 1081011780 seconds
after the 'epoch' - 1/1/1970 - in Unix terms.

I grant you that it can be useful to use a text editor on the
files, but that is not reason, at least in my opinion, to add
record delimeters to records which are already defined as fixed
length.

> > On the old 3B2 I remember it was using the largest 5.25" HD drives
> > available at that time - 70MB CDC drives.  If you needed larger
> > capacity your only alternative was the 8" hard drives. This was
> > because the maximum amount of data on a disk platter was limited by
> > the technology of the day.

> You were running Oracle on a *3B2*??

No - this was a filePro installation if migrated from Profile HD ??
[was that the name] on a Model III, to a Radio Shack 16, to
the AT&T 3B2 with the exteral cabinet for HDs.  It didn't seem too
long before the client filled those and moved to a system with
three 200MB HDs stripped in a RAID to be able to get the required
amount of space.  As it was the 200MB HS's were about $700 each.

That seemed large for it's time, but the poster who started what
has become this thread was having problem with xfer and his FP
database has 30GB of data.  I don't know how large his biggest file
is, but they are probably substantial.   I know when I've brought
500-700MB >files< into a text editor [using what I had available at
the time] things get a bit tedious as the files are read in,
buffers are filled, etc.

> > But as I look at things changing things for reasons such as 'easier
> > to use', 'use standard tools' 'everyone else does it like that' is
> > always the successful way.   My POV is that if you are going to
> > change don't follow the world, find a better way of doing things,
> > as being a copycat is not usually the way to financial success.

> Depends on your goals.

If you want to be a follower that is find.  But when you follow and
the leader and just adopt their changes there then starts to be
little reason to stay with what you have and not go to the leader.


> 
> Cheers,
> -- jra
> -- 
> Jay R. Ashworth                                                jra at baylink.com
> Member of the Technical Staff     Baylink                             RFC 2100
> The Suncoast Freenet         The Things I Think
> Tampa Bay, Florida        http://baylink.pitas.com             +1 727 647 1274
> 
>         "They had engineers in my day, too."  -- Perry Vance Nelson
> _______________________________________________
> Filepro-list mailing list
> Filepro-list at lists.celestial.com
> http://mailman.celestial.com/mailman/listinfo/filepro-list
> 

-- 
Bill Vermillion - bv @ wjv . com