[ale] "Back-End" Filesystems

[Joseph A. Knapka]

Jeff Hubbs wrote:
> 
> I have a question that's so off-the-wall that I'm having trouble figuring
> out how to ask it!
> 
> I can wrap my head around the idea of writng a piece of code (say, in Perl)
> that takes a file as standard input and munges it in some way, producing a
> result to standard output.  Qrpff, the seven-lines-of-Perl DVD descrambler,
> does just that.
> 
> My question - best as I'm able to phrase it - is this:   is there a way to
> substitute writing a file to a point in a filesystem for directing a file to
> stdin as described above?
> 
> Using the qrpff example (I'm not interested in using qrpff in reality but
> it's close in concept to what I'm after), I would say
> 
>         cat file.vob | perl -l
> <colon_delimited_bytewise_decimal_of_DVD_title_key> qrpff | <etc...>
> 
> to feed a file (in this case, DVD content) to qrpff.  Instead, imagine that
> I want to say
> 
>         cp file.vob /mnt/qrpff_input

I'm with you so far, and timball's suggestion ("use a fifo") is
adequate to this point.
 
> Note that I'm alluding to something a bit more involved than just creating
> some kind of character or block device such that I would have said:
> 
>         dd if=file.vob of=/dev/qrpff_input
> 
> Rather, I'm trying to create some kind of virtual file system which, I
> presume, would be mounted in much the same way as one uses the loopback
> device to mount an .iso file as a read-only file system which can be
> accessed as though it were part of the system's own tree.
>  Similarly, this
> file system would have to handle reads, queries (think "ls"), and all the
> other filesystem things we're used to, like user and group ownership,
> attributes, and directories (maybe links as well).

Now you've lost me. You want to send input to some process via
a file-like interface, fine. But you also want something more,
and that something is... to have the receiving process catalog
the input in a filesystem-like manner? I don't think I really
understand what you're getting at here.
 
> Can someone speak to what this would be like?  Would it be a little like
> trying to implement reiserfs or iso9660 - i.e., writing a module?  Would I
> be right to presume that the Perl, Python, or whatever would have to be
> explicitly programmed to respond to (what I presume are) library calls to
> file systems in the proper way?  Please bear in mind that I'm a little
> hamstrung by not knowing exactly what happens when I type "ls -l" (I know, I
> probably ought to just RTFS).
> 
> - Jeff

What happens when you type "ls" is essentially the following:

(1) The shell (eg Bash) parses the command, figures out that it's
the name of an executable file ("/bin/ls"), and invokes a system call
to ask the kernel to execute the file.

(2) The kernel creates a new process, with its own virtual memory,
arranges for any command-line arguments to be put someplace the new
process can find them, and arranges for its standard input and output
to be hooked up to your shell process in the appropriate manner. It then
loads the /bin/ls executable into the new process's virtual memory
space and begins executing it.

(3) /bin/ls makes some system calls into the kernel to fetch the
directory information for whatever directory you specified on the "ls"
command line.

(4) The kernel does "whatever is necessary" to fetch the desired
info. Normally that will involve calls into the kernel's Virtual
Filesystem code, which provides a uniform interface to all the
filesystems Linux supports. The "ls" program, however, doesn't care
about that at all. All it cares about is that it can make a system
call, and the kernel will magically provide the requested data.

(5) "ls" writes the directory info on its standard output, which
is received by your shell and printed on the shell window (all of
which itself involves a whole bunch of kernel interaction).

The great thing about this is that all "ls" has to know about is
the kernel system call interface (and really it's libc that
knows about the syscall interface, which is hardware-specific --
userland programs just call libc functions to interact with the
kernel). This means that all the different filesystems behave
identically as far as "ls" is concerned.

So if you can implement the functionality you want as an
actual filesystem (which it's still not clear would be the
right thing), then all normal file operations - read, write,
directory search, redirection, pipes - will work.

Does that help at all?

-- Joe Knapka
"You know how many remote castles there are along the gorges? You
 can't MOVE for remote castles!" -- Lu Tze re. Uberwald
// Linux MM Documentation in progress:
// http://home.earthlink.net/~jknapka/linux-mm/vmoutline.html
2nd Lbl A + 1 = 2nd Pause 2nd Prt GTO 2 R/S
--
To unsubscribe: mail majordomo at ale.org with "unsubscribe ale" in message body.