[ale] need some Evolution email magic

[Michael B. Trausch]

On Fri, 2011-01-21 at 10:18 -0500, Ron Frazier wrote:
> Hi all,
> 
> I have two scenarios that I need help with re evolution email, which I
> just recently converted to.  The scenarios are listed below.

I will address these below.

> I've heard comments both online and from some local people that
> Evolution has some serious bugs.  I believe it lost 1000 or so of my
> messages (from mailing lists), but I couldn't possibly tell you what I
> was doing or how it happened.  Since then, it seems to be working OK.

I can at least say that I have not run into any problems with Evolution
dropping any of my mail on the floor, so to speak.  But, I do 99% of my
email management manually, and I try my best to always leave my INBOX
with zero messages when I am done processing mail.  I fully process my
mail at least once per day, and more frequently than not, twice or more.
But even if I don't get to mail management during the day, I (with some
exceptions) always fully process my mail just before I go to bed each
day.

As far as keeping my INBOX at zero messages, here is what I do:

 * Do:  If a message requires my attention and I can do it in a small
   time frame (say, three minutes or less) then I do it.  Then either
   archive or delete.

 * Delegate:  If a message requires the action of someone else, then
   I forward it along.  Then, if it needs to be archived, I will put
   it in the appropriate folder, or if not, I just delete it.

 * Defer:  If a message requires my attention but I cannot give it the
   attention that it needs in three minutes or less, I defer it.  When
   I do that, I leave it in my INBOX, and I mark it with the Important
   flag so that it sticks out and I don't delete or archive it before
   I am finished with it.

 * Delete:  If a message does not require a reply, and I have no need
   to archive it, I delete it as soon as I read it.  Sometimes I even
   delete it before I read it, depending on origin and subject.

I explicitly do not have any automatic filters on my messages, even for
my massive mailing list messages.  And I try very hard to keep my folder
sizes reasonable.  If a folder is too large, it can take a while to
access the folder and manage it.  To help achieve that goal, I regularly
purge my Sent mail folder into an archive (e.g., Sent/2011, or if there
are more than 6,000 or so messages in it, I will break it up into
2011/1, 2011/2, etc.).  I prune other folders as the need arises.

> 1) I have two computers, Dell and TAZ, both downloading messages from my
> separate but duplicate pop accounts.  POP account 1 copies all its
> messages to POP account 2.  I want to have duplicate messages on each
> computer, and I want them stored locally, which is why I'm not using
> IMAP.  TAZ has been running evolution slightly longer than Dell, and has
> more messages accumulated.
> 
>      A) I want to archive all the TAZ messages that are older than the
> ones on Dell, then erase them from TAZ.  I have to be able to get them
> back later if desired.  I get LOTS of messages from mailing lists.  I
> have 13,000+ in my mailing lists folder on Dell since Dec 8.  TAZ has
> more, maybe twice as many.
> 
>      B) I spent a good deal of time customizing the folders on Dell, as
> well as filters.  I want to erase the evolution configuration on TAZ,
> and then duplicate the configuration on Dell to TAZ, except that TAZ
> logs into a different email account.  If I can, I want to duplicate the
> messages and their folder locations from Dell to TAZ.  If not, I at
> least want to duplicate the folders, filters, and configuration from
> Dell to TAZ.
> 
>      C) Once I get the two computers synchronized with the same
> messages, filters, and folders, and configuration, then they can each go
> about their business, downloading mail.  I don't care if they remain
> totally synchronized or not.  I just would rather not have to recreate
> all the filters and folders on the second system.

This scenario is best serviced by using IMAP.

One thing that you will notice (on any sort of system, really) is that
having so many messages in a single folder will reduce the performance
of the MUA, usually noticeably.  You can avoid that by keeping the
number of messages in a folder to less than (approximately) five to ten
thousand.  Read on for why.

Mail servers store mail in one of three ways:

  1. On the filesystem, in a sequential access text file (such as the
     ancient UNIX mbox format).

  2. On the filesystem, in a directory tree with one file per message,
     and each message is in a directory associated with an IMAP folder.

  3. In a database, with one of any number of layouts.

For the first situation, performance can begin to suffer with as few as
a thousand messages in the "folder" (file).  This can be mitigated by
building and maintaining an index, so that offsets of messages are known
based on their sequence, subject, date, or whatever the index is built
to determine.  If the index is corrupt or missing, then all operations
on the mbox file will require scanning the file sequentially from the
beginning; e.g., if you want the 3,500th message in the file, you have
to read messages 1 through 3499 in order to get to it.  Bad, bad.

For the second situation, performance depends on the filesystem.  For
very old filesystems, performance will royally stink beyond a few
thousand messages.  Even for newer filesystems, having 10,000+ messages
in a directory can cause the directory itself to grow to use a lot of
storage, and the time required to access a single file in that directory
can grow to cause a noticeable delay.  This can be mitigated by ensuring
that such directory trees are located on filesystems that handle this
type of problem efficiently.  ext4 does okay, btrfs is supposed to do
awesome, but btrfs still has problems (as of kernel 2.6.35) that make it
unsuitable for production use yet (including very slow I/O when using
any sort of software that makes frequent use of the fsync() system
call).

For the third situation, access efficiency depends on whether the
database is properly setup (normalized and indexed).  But in theory it
should scale pretty well, since databases are designed to hold a great
deal more data than either mbox or maildir (or anything like either of
them).  I have no personal experience with this setup and storing huge
numbers of messages (yet).

> 2) I get a number of routine messages from senders that I've given my
> address, such as Verizon (ie - your bill is ready, etc.).  I want them
> to come into my inbox, and I want to see them.  However, after I've seen
> them, I no longer need them in the inbox in general.  On a typical day,
> depending on the time of the month, I might get mails from Verizon,
> Geico, Borders, Barnes and Noble, Best Buy, Solar Thermal Magazine, etc.
> There may by 5-15 per day.  After a few days, the inbox gets crowded.
> However, it's quite a pain to move each message to it's individual
> folder every day.

As I mentioned above, I do manage this by hand.

This is precisely the type of task that an embedded interpreter would be
useful for.  Actually, I don't know why this idea did not occur to me
sooner.  I should look to see if something like this has been done;
Evolution has a well-written plugin system.  It should be possible to
create a plugin for Evolution where you can have little Python scripts
that do very specific things, and have them bound to keys, so that you
can have macros for things like "move message to folder" or "run the
filters for this single message", and have things like manually
triggered filters.

I will look into this sometime in the next two weeks.  I think this
would be something worth spending my time on if it does not already
exist somewhere; it'd be a huge time saver for me (and I suspect others,
as well).

> I want to set up filters for each message that will move them to their
> proper folders.  However, I only want the filters to activate when I
> manually select the messages and start the filter.  I do not want the
> filters to run automatically on incoming mail.  So, every two weeks or
> so, I would select all the read messages hanging around in the inbox,
> and activate the filter, and they would vanish to their specific
> folders, if rules are defined for each individual message.  Any messages
> which did not have rules would remain.   On Eudora, I could set up rules
> for incoming, outgoing, or manual modes, and any combination of them.  I
> haven't figured out how to do that with Evolution.

Same as last ¶.

> What would be even better, is if I could have the filter run
> automatically each day, and run only against all the read messages over
> 1-2 weeks old.  Then, I never have to touch it.  I specifically do not
> want my incoming mail filters to run in this context, only the manually
> activated filters.

The only way I could think of to do this while using POP would be to
have something in your user session running that killed Evolution at a
certain time every day, manipulated the data files directly, removed the
indexes (and forced Evolution to rebuild them) and then restarted
Evolution.  Upon restart, Evolution would reindex all of the folders and
resume normal operations.

For IMAP, you could use a small script and just have it done once per
day.  But it would be something of a pain, I think, since running
filters in that way can take way longer than expected, particularly if
the IMAP server is slow and/or far away (network-wise).  And some IMAP
servers are pretty quirky, they'll hang up after being connected for so
long or after so many bytes have been transferred or whatever.

	--- Mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://mail.ale.org/pipermail/ale/attachments/20110122/644a176e/attachment.bin