[ale] Filed De-duplication

[John Pilman]

David Hamm posted to this list about three years ago:
If you are trying to track down all the duplicate files in each clients
directory I'd suggest fdupes.
Fdupes will search a directory tree for duplicates by generating an MD5 sum
of
the contents of each file and offers options for dealing with duplicates.
It
will produce a list, or even remove any duplicates and create links to the
remaining file while preserving the file name.

I have been using fdupes to llist duplicates to a file and I usually delete
the diplicate with the longest path.

...John



On Fri, Oct 18, 2013 at 12:59 PM, JD <jdp at algoloma.com> wrote:

> Slashdot had a question about this 1-2 yrs ago.  Lots of people suggested
> scripting it, others pointed out some C code on sourceforge.
>
> I had a few hrs free that day and wrote some Perl (200+ LOC). Use it all
> the
> time, but I'd probably go with the C tool for any very large datasets.
>  Mine
> doesn't automaticly remove anything and is far from perfect, that is
> certain.
> It is relatively fast on most types of files, however.
>
> On 10/18/2013 12:34 PM, Calvin Harrigan wrote:
> > Good Afternoon,
> >     I'm looking for a little advice/recommendation on file de-duplication
> > software. I've have a disk filled with files that most certainly have
> > duplicates.  What's the best way to get rid of the duplicates.  I'd like
> to
> > check deeper than just file name/date/size.  If possible I'd like to
> check
> > content (checksum?).  Are you aware of anything like that?  Linux or
> windows is
> > fine.  Thanks
> > _______________________________
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20131018/18801d76/attachment-0001.html>