OT: multi-platform *nix blocksize detection

[Bill Vermillion]

On Wed, Oct 20 17:49 , Fairlight, showing utter disregard for 
spell-checkers gave us this: 

> Only Bill Vermillion would say something like:

> > But you are such a good strait man I couldn't resist.

> If I'm such a strait man, why do I feel like I've lost my Bering?

You're being cool for not catigating me on the spelling error.

> > > True. Which is -why- I wanted a surefire way to find the
> > > default block size--preferably without relying on 'dd' and 'ls'
> > > to do so. It's tempting to look inside the source to 'ls' and
> > > find out exactly how it comes up with its number of blocks. It
> > > seems to be reliable on any system.

> > You can NOT rely on 'ls' because it will show you the length of
> > the file based on the information in the inode by looking at the
> > starting block, the ending block, and the ending byte in the last
> > block. The actual disk space could be far far less than that.

> ls -ls 

> -s    Gives size in blocks, including indirect  blocks,  for
>       each entry.

I'll accept that.


> > > As for your second question...yes, the operation should be
> > > filesystem-dependant, although you have to keep in mind that
> > > the VFS layer is in the kernel, and that's probably where
> > > linux's bits/stat.h comment about 512-byte blocks for st_blocks
> > > comes into play.

> > I disagree.  I would think it should be filesystem-INdependant.
> > That way you should be able to use it among all OSes extant.

> I meant to say it -should- be INdependant, and therefore the
> "although" would have actually come into play. That's what I
> get for typing, being on the phone, and coding in other windows
> on and off all at once.

Ah - one for one.  INdependant vs strait.

> > Yup.  Even SCO stores very short files in the inode.  It makes good
> > sense as it save an extra disk seek and therefore speeds up
> > performance.  This is perfect for symlinks. 

> Okay, that makes sense. Although I've seen symlinks take 2
> blocks before. I saw one today, in fact. I suspect that in
> reality, it's showing the block count of the resolved link
> target. But that's one area where 'ls' is hazy.

I have some nagging memory that the SCO stores files in the inode
if they are under 56 bytes long.  That's from an old release note
as I recall.

--------------
>From    stat(2)  on FreeBSD  [Glimpse is really handy]

     st_blocks	    The actual number of blocks allocated for the file in
		    512-byte units.  As short symbolic links are stored in the
		    inode, this number may be zero.

----------------

I guess you'd have to read the code to find out more.


> > And what about a system that has different block sizes and perhaps
> > different filesystems on the same drive which it can access.

> In -theory-, you should have been able to use st_blksize specifically for
> this issue.  I don't know if it's just linux that does it incorrectly or if
> other *nixen do it wrong as well, but no matter which system I'm on, or
> which filesystem or kernel, it's set to 4096, which is -not- the block
> size.  Of that I'm sure on the systems I've tested.  (This time I really am.)

>From the above it sounds as if it should work.  Have you tried this
on more than one Linux system, and what was the filesystem type?

...

> > Then the only thing that is going to let you how much space is
> > being used AFAIK is 'dd' if that's what you are hunting for?
> 
> dd shouldn't work.  man lseek:
> 
>        The lseek function allows the file offset to be set beyond
>        the end of the existing end-of-file of the file.  If  data
>        is  later  written  at this point, subsequent reads of the
>        data in the gap return bytes of zeros (until data is actu-
>        ally written into the gap).

Score:  Mark -1, Bill -2.

I >meant<  'df'.   That makes the rest of your comments on dd
not applicable.

> So in essence, if you dd from the file, you should end up with
> either the full size of the file if a segment was written at
> the end, or a truncated length that's inflated beyond what's
> actually contained in the file, as there will be 10MB+ gaps in
> the file in some instances--sometimes concurrently if a mesh
> came in poorly. Wholly useless.

But now looking at 'dd'  if you use the -spare flag to dd it
appears it will skip outputting NUL.

> > The problem I can see is that if the file extends the file to the
> > full length and there is not enought space on the drive when all
> > pieces/parts are retreived you will have a corrupt file.

> That's why you watch your disk space, or just keep large drives.  :)

And that reminded me of a desgin flaw that I've seen bite more than one
person in MS land.   If you attempt to write a file that will fill
the drive, the write or copy will stop and that file we be removed
totally.

I had a friend complain about a bug in MS when he had loaded a file
from disk, modified by making it longer, long enough to use up the
remaining space, so when he wrote the file out, and the system
noticed that it was too long to fit, it removed the entire file.
He had no backup on it.

In *n*x you will write up to the poing of filling the disk but the
data will still remain.  Maybe that's has been changed with the
NTFS system.

But at least in *n*x when you suck a large file in you will keep
the data up to the point when the disk fills.  In a multi-stream
environment if you are writing from multiple sources/pieces, I see
that you could get a file with parts not being all there perhaps in
the middle.  If it you just lost data at the end of the file you
could possibly salvage that.

> > So you could see how big the file is going to be via an 'ls' but
> > you'd have to use something like 'dd' to look as the amount of 
> > real space available.   But that is going to be slow.

> But 'dd won't (or shouldn't) work accurately, for the reasons I cited
> above.

But 'df' should.  The only problem with df is that it will show you
the disk space used, and if your OS uses large caches dd won't show
you the true useage unless you can flush the buffer. Perfoming
fsck on a mounted system will run in the no-write mode but will
flush the buffers.

> > I take it this is going to be for something like bittorrent?

> It's for gtk=gnutella, actually. Same concept. Although I
> note that I have GetRight for win95 and it's just a http/ftp
> download program that uses multi-segmented downloading for
> download accelleration, which uses the same technique.
> (It's nice when you get sites that limit you to 40K/sec per
> connection and you can just toss four or six segments at it,
> thus getting it at full blast.)

And if the pieces/parts are stored separately and then just merged
into one contiguous file it would look just like the old indexing
on the early filePro - where you could watch the indivual pieces of
the index files being built, and when all data was sorted into
these little piles, it would look at each 'pile and get the next
correct byte building the index.

One thing about the old/slow machines in the past is that you could
watch the operation of the system and get a good feel of just how
things worked.  Now things are so fast you don't have that
opportunity.

> > Is it checking for true free space?  If so you might snag their
> > code and check it.  It uses the MIT License so you can do almost
> > anything you want with it.

> Well, gtkg is GPL'd, I believe. I could look through it...I
> have the last three trees sitting here. I don't believe it
> does check for the full real space when creating and extending
> the file--and it would be useless if it did. It would only be
> useful if it performs a check for free space before starting
> each segment, as not only other files may have been altered and
> grown in the meantime, but other applications could have sucked
> up a bunch of space--and it can't track all of that just based
> on a single check at file inception.

Aren't you getting those segments in parallel?  If so even checking
before each segment was being loaded in would still mean it
wouldn't work if anything else is happening.

Assuming that the application had control of the OS which had to
wait for the ap to finish was one of those things that drove me to
Unix 20+ years ago.

Bill
-- 
Bill Vermillion - bv @ wjv . com