[ale] School Project to Create Distributed Filesystem

[James Taylor]

One other one that looks interesting.

http://www.scs.stanford.edu/shark/

-jt
 
 

James Taylor
The East Cobb Group, Inc.
678-697-9420
james.taylor at eastcobbgroup.com
http://www.eastcobbgroup.com







>>> Greg Freemyer <greg.freemyer at gmail.com> 2/24/2009 10:57 AM >>> 
I just took a look at your goals.  Very admirable.

I don't think they is any practical way a school project could develop
a distributed filesystem like that from scratch.  (Unless you have
many man-years to invest).

Fortunately there are several existing  projects you could likely
leverage.  I suggest you research them then ask some basic
architectural questions here as you try to create your basic design.

First what you are calling a distributed filesystem I normally call a
cluster filesystem, and I think many in the industry do as well so
keep that in mind as you read.

Below is an initial list of OPENSOURCE projects you should at least
look at briefly.  I think all of these are implemented in Linux and
have source code available.  I believe only UnionFS is in the vanilla
kernel, so you'll need to hunt down the patches if you want to see the
code for any of the other projects.

1) GFS from Redhat.  A cluster FS.
2) OCF2 from Oracle. A cluster FS
3) ZFS a filesystem from Sun.  Offers mirroring I think.  distributed?
4) BTRFS another new generation FS with checksumming.  Not sure if it
has any relevant aspects or not.
5) DRBD from linbit(?) - a distributed raid-1 kernel patch. ie. half
the mirror on one computer, half on the other.
5.1) Linux-HA - cluster enabling code typically used with drbd.  Has a
shiny new drbd/linux-ha gui available (as of a couple weeks ago).
6) CFS from OpenSSI - a cluster FS
7) UnionFS - A layered filesystem the sits above other FS's. I would
look hard at this if I were you.
8) Lustre - A true distributed FS that may satisfy all of your goals
already.  So maybe you want to avoid Lustre?  Or if just a goal or two
are missing, maybe you could enhance it?

All of the above are currently functional.  Most are production quality.

Many of the above will be useless to you, but you should at least
review their feature sets before you ignore them.

I'm sure I missed a few.  The above is from the top of my head.

And as Mike said, stick to FuSE for as much work as you can.  Working
on the linux kernel filesystem layer is very complex and not where I
would want to start my journey in to kernel hacking.

On Mon, Feb 23, 2009 at 11:40 PM, Omar Chanouha <ofosho at gatech.edu> wrote:
> Hello Everyone,
>
>       I have a school project at gatech this semster that involves
> creating a distributed filesystem. The direct link to our proposal can
> be found here:http://www.opensourceaficionado.com/winabi/tiki-download_file.php?fileId=5.
> The website can be found here:
> http://www.opensourceaficionado.com/winabi.
>
> Essentially, what we want to do is create a distibuted filesystem via
> a protocol we will create, but connect to it via nfs, so the client
> does not need to have any special software on his/her pc. The section
> I am assigned to is to create the nfs to winabi interface. Below is a
> short example:
>
> client ---NFS--->server--->winabi--->filesystem
> client <---NFS---server<---winabi<---filesystem
>
> I would like ANY help a kind soul is willing to offer. A few questions
> that would be nice to have answered, but again, any thoughts/comments
> you have would be appreciated.
>
> 1. Would modifying using space apps alone, ie. nfsutils, be enough?
> (Probably not, right?)
> 2. Is there any documentation, I am having a little trouble finding
> it? Specifically,
>   -What comes with nfs-utils, and what do they do?
>   -What is in the kernel, and what does it do?
> 3. If I have to modify kernel code, where is that code located?
>   -If anyone could give me specific files to change, you would be my hero
>
> Also, no one on our team has done any kernel hacking before, so any
> advice AT ALL as related to kernel hacking in general is appreciated.
> Any wisdoms you may have learned, anything you would have like to know
> when you were getting started, any opinion at all is welcome.
>
> Lastly, if anyone thinks that they have alot to share in terms of
> kernel hacking or nfs and would like to pass on some skills, I got
> dinner/lunch whenever you like.
>
> Thanks alot!
>
> -OFosho, Miami Dolphin and Linux Enthusiast
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
>



-- 
Greg Freemyer
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com

_______________________________________________
Ale mailing list
Ale at ale.org
http://mail.ale.org/mailman/listinfo/ale