Disk caching and data reading (was Re: create())

[Kenneth Brody]

On 9/6/2013 7:57 PM, Fairlight wrote:
> On Fri, Sep 06, 2013 at 11:37:23AM -0400, Kenneth Brody thus spoke:
>> It doesn't matter when (or even "if", for the case at hand) the data is
>> written from the cache to the physical device.  As soon as the data is
>> placed in the cache (which will happen prior to any[*] write call), it is
>> available to be read by any other program.
>>
>> There seems to be a misconception among some people that reads will
>> somehow circumvent any write cache, and that modified data won't be seen
>> until the cache is flushed to the physical device.  This is simply not
>> true -- if the data is already in cache, then any[**] read will fetch the
>> data from that cache.
>
> I always looked at it as the cache being FIFO, but it sounds like, from
> what you're saying, it basically presents a FIFO-like appearance, even when
> it's not purely FIFO.  Either way, I agree with you that it should not be
> problematic.

Whether or not a write cache is physically written to the HD in FIFO order 
or not, a cache is not FIFO, nor any other order.  (If you need a name, it 
would be "random access".)  The cache is simply "RAM is faster than HD, so 
let's keep things in RAM when possible".  For writing, it means that the 
program doesn't have to wait for the data to be physically written to the 
HD.  (And the O/S will eventually get around to writing it to disk.)  For 
reads, it means that, if the data is already in the cache, it is available 
immediately.  (And read-ahead cache can mean that it's already in RAM, even 
if it was never accessed until now.)  Otherwise, it must wait until it gets 
around to reading it from the HD.

The point being, if you write new data, and it's still in the cache and not 
on the physical device, if something else tries to read that data, it will 
fetch the new data from the cache, rather than attempt to re-read the 
physical device and get the old data.

>> The only problems I am aware of with directories with lots of files is
>> slower open/close functions.  Once open, it doesn't matter how many files
>> are in that directory.  (In fact, on *nix systems, you can even delete
>> the open file itself, and continue using it.)
>
> Open/close on files in heavily populated directories should not be slow.

If a directory has 2000 files in it, especially if the directory itself is 
fragmented, then (depending on filesystem type) it may require numerous disk 
reads before the directory entry is found.  (On some filesystems, if the 
file is the 2000th entry in the directory, the O/S needs to read all 2000 
entries and do a string compare on each of them before it will find the 
correct entry.)

> The only operation I know of, at least on *nix systems, which is adversely
> affected by heavily (>500 files) populated directories is readdir().
> That's due to double inode redirection being triggered over about that
> point.

The file must first be found (probably using the kernel's readdir 
equivalent) before it can be opened.

However, once opened, the number of files in the directory is irrelevant to 
any I/O on that file.

[...]
>>> If you're chasing a virus problem, empty the temp folder. Some
>>> antivirus software delete temp stuff too.
>>
>> I would call such an AV program "horribly broken".  Unless it detects a
>> virus in one of those files, it has no business touching *anything*.
>
> Any A/V program that automagically deletes files without requiring approval
> of the user should immediately be removed from the system.

Many (most?) AV programs have the *option* to do such things automatically. 
  Whether that option is on by default, I don't know.

>>> If MS just 'fixed what we have' and called it a new OS, I'd buy.
>>
>> I find it hard to believe that Windows is "broken" in this regard.  If
>> it were, it couldn't function.  And, in my more than 25 years experience
>> with Windows, I personally have never seen the situation described.
>
> I haven't either.  I mean no offence to anyone (though some will doubtless
> be taken or construed), but sometimes I think people just make this stuff
> up at random, or hallucinate it.  Then it gets related in fora such as
> this, and attains the status of virtual urban legend.

-- 
Kenneth Brody