logfile filesystem…
I would really like a logfile filesystem. Something like FUSE could do this well.
I was thinking of this this morning. Managing logs from server applications is a real pain. There are numerous problems. I’ll use Apache as an example, but this applies to almost everything.
The first pain is log rotation. When Apache, say, is appending to a log file, you need to get it to close and re-open the file in order to successfully rotate it. This is difficult to do cleanly—no matter how you do it, Apache notices. There are two “traditional” methods to rotate Apache logfiles. (This applies to other server processes as well, although with different levels of rotation support in the application.)
You can rename the “live” logfile, while Apache’s still writing to it. Then tell Apache to close and re-open its logfiles (thereby creating a new one at the old name). This ensures no logs are lost, but interrupts traffic. Apache, specifically, supports a “graceful” restart option that shouldn’t drop connections, but this, in turn, can be problematic: there’s no simple, cross-platform way to know when those connections have finished and the old logfile is closed, and there are some cases where code running in Apache as a module can fail to work correctly after such a restart.
Or, you can tell Apache to log to a command over a pipe. The theory then being that these processes handle the log rotation themselves, re-opening files at the appropriate times, with the appropriate names. This actually works really well on small sites, but can be a performance or reliability issue in the long run. It also results in a huge number of these additional processes having to run, one for each log file.
And I’ve fought with both in the past, and been rather unsatisfied.
What really needs to happen, from a reliability standpoint, is for Apache (or whatever server process) to use the simplest possible logging method: open the log file for appending, creating it if it doesn’t exist, and write to it. This is not prone to failure (until the filesystem fills up.)
If there’s a system outage, there will be far fewer log entries still in RAM. Very little CPU is used. Everyone wins.
What’s missing is a file operation that’s the opposite of ftruncate(). There needs to be a way for another, unrelated process to follow along in the file, and move the records to their final destination. And this process needs to be able to delete them from the _beginning_ of the file as it goes.
The second big problem with logfile management is security. It’d be very desirable to easily enforce the idea that an application can only append to its logfiles, never delete, truncate, or overwrite them. This prevents a remote execution exploit from covering its own tracks.
Many OS’s support filesystem flags or ACLs to enforce such a policy, but it’s not portable, and interferes with the usual log rotation methods.
And this is where FUSE comes in.
A FUSE filesystem could be implemented as a _mostly_ passthrough filesystem. It could easily enforce the append-only policy in the presented filesystem.
And one way to implement a correct and complete log rotation policy would be to map the published filesystem’s files to a series of small (16KB-16MB depending on the throughput of logging that occurs) “buffer” files. As each file reaches its maximum size, the next write would go to a new file.
And a separate process could read these files and write log entries to their appropriate “final” locations.
Perhaps a sqlite database could be used by that process to record state (ensuring in the event of a crash a reasonable chance of recovering where it left off).
And the FUSE filessytem’s code path would, for the important case (”append to a file”) be very straightforward.
This requires a lot more thought…
