Too Many Open Files (was Re: The CLOSE command)

[Fairlight]

At Fri, Feb 17, 2006 at 05:42:46PM -0800 or thereabouts, 
suspect Jeff Harrison was observed uttering:
> 
> No its not.  One should always use the -p flag on
> lookups that could eventually be written to.  Filepro
> will automatically place a lock on the file - if one
> is not there already - when an actual assignment takes
> place, but this may be too late if someone has in the
> meantime grabbed that record.  Using -p flag prevents
> this possibility.

What happens if it's "too late" and someone already grabbed that record?
The fact that it protects a record semi-automagically only makes this case
more ambiguous--at least to me.

> This is stupid.  You are going to need to look to the
> parts file and the customer file again and again.  Why
> close and re-open it all the time?  This would be VERY
> inneficient.

I agree.  But you -should- close it at the first atomic moment you're done
with the -batch- of lookups then, if you have further processing to do--not
just let them hang open until you're done.

> So what?  You have a few more files open than your need.  I believe that
> the alternative of too many opens and closes is far more costly.  Again,
> filepro will close the files as needed.  If it does not, then it is a bug
> in my opinion and should be fixed.

The fact that you ask, "So what?" only tells me you missed my point
entirely.  I can only assume you are likely an application rather than
systems person.  Okay, I'll break it down for you and the other non-systems
folks, then.

Example:  You have a process that uses 10 external files.  Only 2 of these
actually -need- to be open simultaneously.  Assume you don't close your
files.  That's 10 file descriptors you've eaten through.  Add the base
three fd's (STDIN/STDOUT/STDERR), and you're up to 13.  Per process.

Now multiply by instances of said program running.  If you have 5 processes
of -just- this kind running, that's 65 file descriptors.

Are you -aware- that some OSes default to a 60fd PER USER fd limit?  That's
across all concurrent login sessions for the same user, not just one
individual login session.

This doesn't even take into account any other processes you may have
running.  Your shell (for EACH login session) takes three fd's minimum.
Any other processes you run also take a minimum of three fd's unless they
close them immediately.

There's a tunable system-wide fd limit in linux.  Add a web server, a
bunch of CGI programs and their children, maybe an IRC daemon or any other
server-type service (ssh is a good example of one that's likely to be
running), and you push the resources pretty hard.  Now if you can -slice-
that fd use by 80% where you can by using close(), you won't be nearly as
likely to run into problems.

I can't remember what, if any, per-user limits there are on linux, much
less other OS'es.  At a glance, I can't find it in the linux include files
where I'd expect it.  I remember BSD 4.3 was tuneable.  So was SCO (3.2.4.x
and 5.0.x).

I also am unsure of one thing in specific--and this could make a huge
difference:  I don't know if the fd count in a per-user scenario is based
on UID or EUID.  I should think it would be by UID, but it may depend on
a few things, some of which may be OS-specific.  I don't consider myself
qualified enough to give near a definitive answer even on linux without
testing.  In fact, I'm not sure how to test "who owns this fd?" at all,
to be honest, short of trying to push the limit.  But if it were by EUID,
then if all those get accrued to the filepro user, you can get up to the
limit far more easily.  I suspect it's not, but as I said I can't prove it
easily.

Either way, system resources, globally and/or at the system level are
-limited-.  Polite application developers take as few of them as necessary
at every possible point.  It keeps not only your own app, but the whole
user experience for -everyone- on the system running as smoothly as
possible.  You really don't want to be pushing the limits of how many
people can log in or access the web server just because someone is too lazy
to close fd's when they both have the choice and it makes sense to.

-THAT- is "so what".

mark->