OT: multi-platform *nix blocksize detection

[Bill Vermillion]

On Wed, Oct 20 13:03 , while impersonating an expert on the internet, 
Fairlight sent this to stdout: 

> This public service announcement was brought to you by Bill Vermillion:
> > 
> > Don't confuse physical block size with filesystem block size.

> Point taken.

> > I've just looked at mkfs.ext2 and Linus also allocates in
> > 1024, 2048 or 4096 byte blocks.   So you can not depend
> > on the file system allocation size being the same on all system -
> > but the default will the 1024.

> You know, that's the odd thing. If I look at ls -ls on a file
> on this box, I get one 1K block per <= 1K of data. It's 1024K
> blocks. For example, /etc/profile will show as 766 bytes and
> I get '1' in the ls -ls listing, but if I access st_blocks,
> it kicks back 2. That's what originally led me to conclude it
> was 512-byte blocks. BUT...if I go to Solaris and do the same
> thing, /etc/profile is 1403 bytes, and I get 8 whether I do
> ls -ls or whether I access st_blocks. So they have 4K blocks.
> Okay, fine.

> Something is goofy on linux systems.

If thought that since the first Linux system I saw :-)

> I thought it might just be mine, but a 6822-byte /etc/profile
> on SuSE 9.0 results in ls -ls saying it's using 8 blocks, but
> accessing st_blocks gives me 16. On linux I have to divide by
> two to get the correct information. So they're 2K blocks on
> SuSE, but something is screwy on linux with st_blocks in any
> event, no matter which kernel tree (2.2 vs 2.4 between my two
> tests...Cobalt vs SuSE).

I'm looking at a SuSE system now.  On it the /etc/profile
is 6845 bytes long.    ls -ls gives me 6845 bytes but ls -lk
gives me 7 - it has allocated 7K.

Perhaps during install someone change the default block size
in mkfs.ext2  or mkfs.ext3 [or whatever] when the file system
was created.  You can not depend on all systems being installed
with the default block sizes.

The SuSE 9.1 I looked like is not jibing with the SuSE you
are indicating.  And shouldn't the file creation be independant of
the OS, and in particular the system utilities such as ls, df, et
al shouldn't be kernle dependant but upon the filesystem.

> So it's not even a matter of scaling down to 512-byte blocks,
> it's a matter of always dividing by 2 to get the right number.

On that system.  What is there is another block size specified.
Linux permits 1024, 2048, and 4096.  Systems such as FreeBSD
default to 16384.  They do not recommend going above this - but
as I recall the maximum filesystem allocation can be huge.
In the XFS in Iris you can set allocation units to be 1MB long [or
more].  Which is ideal for large media files to keep the files
contiguous on the disk sub-system.

> And I have -no- idea why this should be so.

I'd suggest checking others to ensure that someone has not
changed the default allocation size during install.

> > I know of no Unix[like] system that uses 512 bytes in the
> > file system.

> Remember when I said yesterday, "I guarantee?" I was wrong.
> :-/ Bad logic surrounding the math on my part. I rethought it
> after reading your post, and I did indeed find that it's 1K
> blocks--which left me all the more confused after I rechecked
> my computations.

> > In an effort to improve performance the AT&T SysV2 brought forth
> > the S52 file system.  That allocated 4 blocks at a time - or
> > 2048 bytes.

> Anyone that's ever dealt with NFS knows the value of large
> blocksizes. :)

> > > Checking manually with fdisk isn't something I'd like to
> > > suggest to an end-user, ya know? :)

> > And I'm curious as to why you need to know this, and why a user
> > would need this.  Unless you are working at the lowest OS level
> > you are going to be constrained by the allocation sizes imposed
> > by the file system.

> I wanted to make something portable, really. I wrote it for
> myself, but it could be handy for others. There are programs
> that create files of the entire size they will be when they are
> done downloading. They then write in segments within the file
> while getting the segments from multiple sources, and the only
> way to tell what percentage you approximately have complete
> (and the approximate transfer rate) is to look at the used
> blocks. I wrote a nice little curses-based monitor program for
> percent complete and download rate so I can watch -only- the
> active files in a queue of this type.

I can see the problem with getting it from multiple sources.
But why not look at the lenght of the file as it grows and look at
it in bytes then it should be portable anywhere.   As far as I know
the byte size will be indentical - and that's probably all you can
count on.

...

> Actually, the other puzzler is how someone creates a file of
> an entire size on disk without actually -taking- the full size
> immediately--but letting it report what it will eventually be.

You mean so that an 'ls' will show the size of the file, but
the 'df' will show only the blocks that have been used.

It's been years since I've diddle with these but an lseek will
let you extend a file beyond the current EOF and as you add
data the 'holes' in the middle will be filled.

Bill
-- 
Bill Vermillion - bv @ wjv . com