Courier Maildrop filtering and my two extensions to its capabilities

>>>	Source code changes to Courier Maildrop so a message can be delivered tagged for deletion.
>>>	An external filter program called subjadd which can add a label [Postfix] to the subject line of an email. It will also create a subject line if there was none.
>>>	My experiences with Maildrop's filtering language, and some simple example files.

Maildrop extension: DELTAG - Tagged for Deletion

This is a modification to two?? files in the Maildrop source code.  So you need to recompile Maildrop to be able to use these.  But you probably need to be able to recompile it anyway to set it to work with your preferred mailbox locations.  Please see my discussion of compiling and installing Maildrop via the parent web-mail directory ../.

Why I want to deliver messages to the Inbox tagged for deletion

I want to replicate the functionality of Netscape's (4.77 for Windows) mail filtering system (also, I guess, Mozilla's and Netscape 7's - but when I last did filtering in the client, it was with N4.77) - but to do it entirely at the server.  I am on several dozen mailing lists and so filtering is vital.  Before using Maildrop filtering, I had N4.77 set up to copy messages to the particular mailbox for that mailing list, and to leave the message in the Inbox, but flagged for deletion.   This client-based filtering happened when I "Alt-F M" Netscape to check for new mail.  The headers come to N4.77 and it sends commands back to the IMAP server about where to put the messages and whether to set the "T" tagged for deletion flag for messages which also remain in the Inbox.  But client-based filtering is a pain for a number of reasons.  With Maildrop, I can achieve all this and more on the server.

See more about this on the ../Mozilla-mail/ page, which has an image of what Thunderbird's email index looks like with this arrangement.

The great benefit of this approach to filtering - copying to the target mailbox and putting a copy in the Inbox, tagged for deletion - is that I can see all the activity on my lists without having to look into their various mailboxes.  When I have scanned the "tagged for deletion" messages, (which I can read, print, reply to or untag with the Del key) then I use "Alt-F F" (for Thunderbird)to expunge the mailbox.  Thunderbird tells the IMAP server to delete them all.  

This client-based filtering worked OK, with some restrictions which I won't bother to list, other than the obvious dependency on always using the one PC for all my email!

The advantages of the server based email filtering system, with IMAP mailboxes on the server, using this Maildrop filtering approach include:

1. By filtering each email as it arrives with Maildrop, I remove all dependence on my email client.  This means I can access my account with a web-mail program such as Postman, SqWebMail or IMP from any web browser, SSH to the machine and use Pine or Mutt (in principle - I haven't done this since I didn't find a text-mode client which displayed messages which are tagged for deletion), or  use Thunderbird via IMAP.  I am generally Thunderbird via IMAP over the LAN - but perhaps also from far away.  Generally, I use a web-mail program running on the same server when I am away - then I only need a web browser, not a properly configured Thunderbird.
2. There is no time spent filtering when I connect via a client.
3. Filtering should be very reliable, since it is not affected by client communication problems.
4. Maildrop's filtering language is extremely powerful.
5. With my two modifications/extensions to Maildrop, I can add labels to the subject lines of messages from mailing lists which have no such label.  For instance, some mailing lists are extremely busy, but there is nothing distinctive in its subject lines.  I will be able to add labels like "[Postfix]" to each subject line of the emails in my Inbox so I know which mailing list they belong to.
6. I can implement virus and spam filtering from Maildrop, as documented in a page you can find from the parent directory.
7. I can do what I like with the mail filtering system, including cause the computer to run a program, make a beep etc. when particular messages arrive.
   

The potential disadvantages of the Maildrop filtering approach is:

1. Thunderbird (or any other IMAP based email client, including a web-mail program) does not know that new emails have been put in mailboxes. (The Inbox is fine - just use the normal "Ctrl T for Thunderbird.)  The workaround is to close the window for that mailbox and re-open it.  The best way to do this is to click on the "parent folder" of whatever mailbox is currently in this window.  For the Inbox or other mailboxes on the "first level" this is a folder with the name of the mail server.  For mailboxes in other folders, it is that folder itself.  In either case, the current window becomes selected to a "non-mailbox" and then clicking on the original mailbox causes Thunderbird to look at the mailbox again, so it sees all new messages.  
2. I have to use SSHtelnet to a shell account and use an editor there (Midnight Commander's built in editor) to edit a potentially tricky .mailfilter file, and then watch to see that it is working and has no syntax errors, typically by sending myself a test message.  However, each section of the file, for each mailing list, has a pretty standardised form, and comments are allowed, so I can document what I am doing.
3. Therefore, this form of filtering really needs to be set up and maintained by a technically competent person.  Good filtering is a technically tricky thing anyway, since it depends on minute details of email headers, and the generation of some kind of text which reliably matches some things and not others.

Operational matters based on experience

I have been using this mail filtering system for years now - since 2000 or so - and it is a great success.  

I find that some mailing lists have a plethora of ways in which they can forward email, and so it is necessary to keep refining my filtering rules.  I expected this.  For instance a list which normally sets the "Return-Path: " header reliably may not do this if the message it is forwarding already contains some other "Return-Path: " and or was Cc:ed to the list, rather than sent To: it.  Increasingly there is a List-ID header which is best to use.

There is an important operational matter to consider.  Since emails can arrive at any time (unless, I suppose, I manually halted Postfix for a while) my plan of seeing every bit of list activity via the "Tagged for Deletion" copies in the Inbox can only be reliable if (when using Thunderbird, for instance) :

1. I use Ctrl T to check for the latest received messages.
2. I then look to see if any new ones have arrived which I want to read - and then read them (or I guess undelete them and read them in the Inbox later, or remember to read them in the mailing list box).
3. I use Alt F, F to delete all messages which are tagged for deletion.
4. No messages arrive in the Inbox tagged for deletion between steps 1 and 3.

I discovered this one morning.  There were a few messages tagged for deletion which had arrived the night before, so I thought "OK. lets clear those (Alt F, F)  and then Ctrl T to get the new messages.  But the Alt F, F deletes all messages in the folder which were tagged for deletion, whether Thunderbird had seem them or not.   So I was scratching my head wondering why there had been so little mailing list activity overnight!

The changed code and what it does

Please see this file for changes for the 2003 version of Maildrop 1.5.3: maildrop-1.5.3-DELTAG-mods.tar.gz

Here are the changed files for Maildrop 2.0.4.  

maildir.C is in the maildrop directory.  The unmodified version is 6169 bytes, last changed on 2004-01-15.

maildircreate.c is in the maildir directory.  The unmodified version of this file is 4670 bytes, and hasn't changed since 2003-01-23.  So my modified version hasn't changed since when I did these mods in 2003.  However, the modified version below is a different length since I turned tabs into spaces in 2008.

  maildir.C.txt              9,922 bytes << Modified version  
  maildir.C-orig.txt         6,169 bytes

  maildircreate.c.txt        7,707 bytes << Modified version
  maildircreate.c-orig.txt   4,670 bytes

I had previously done a full install of Maildrop, as documented in another page on  ../mta-integration-01/ .  That was without any of these source code modifications.  Please see the second Maildrop section of that page for how I used these modified files to create a new version of the Maildrop binary which has these capabilities.

In the mail filtering file (.mailfilter in the user's home directory) the filtering commands can be used to set a variable "DELTAG" to "1", any subsequent delivery, either "to" or "cc" which are to Maildir format mailboxes will result in the message having the "T" flag set on its file name.  IMAP servers recognise this as the "Tagged for deletion" flag.  At least with Courier IMAP and Thunderbird, the email will be moved from the /new/ directory of the Maildir to the /cur/ directory, with this tag intact.  Then it will show up in the client as tagged for deletion, and a subsequent IMAP expunge command will delete it.

The source is well commented, self-explanatory and all the changes are tagged "RW".  Copyright public domain.

See the filtering section below for some examples.

subjadd - add a label like [XYZ] to the Subject: line of an email

Here is the source file:

subjadd-c.txt        14,703 bytes

subjadd is a filter - working from stdin to stdout.  Maybe it will be useful for other situations than being called from Maildrop.  Please let me know if you have any such uses.

Here is the usage text for subjadd:

  Adds a label such as [XYZ] to the subject line of an email on
  stdin and sends it to stdout.  The label is the only argument.

  subjadd [XYZ]     results in:

    Subject: Foo                Subject: Re: Bar
    Subject: [XYZ] Foo          Subject: Re: [XYZ] Bar

  Always passes the file, but only writes the label if it is the
  sole argument.  If there was no subject line, then an empty one
  is added, preceded by an informative header line:

    X-Subjadd: No subject line, so this one was added by subjadd.

  "subjadd" primarily means "adding to the subject line"
  but if there was no subject line, then its function is to
  "add a subject line" too.

  Always returns 0.  Intended to be used with the xfilter
  command of Sam Varshavchik's Maildrop mail filtering program.

  Public domain.   Robin Whittle  www.firstpr.com.au  8 July 2001
 

The compilation instructions are in the file:

cc subjadd.c -o subjadd

Exploring Maildrop's filtering

Note: see the parent directory for another page with more Maildrop filtering examples and how I use it to run Spam Assassin and Anomy Sanitizer.

Maildrop has an extensive and well organised filtering language.  The typical arrangement is for each user to have a .mailfilter file in their home directory.  This is a text file, using the commands documented in:

http://www.courier-mta.org/maildrop/maildropfilter.html      Documents the filtering language.  (This is a terse and highly technical language!)

http://www.courier-mta.org/maildrop/maildroptips.html Hints on using the filtering language - old, from 1998.

There is not much in the way of examples in this documentation.

Patterns can be matched, typically in the email's header, but also in the body or in both, and actions taken.  Actions include delivering it to a mailbox, by default to the Inbox, or to any number of email address.  An email can be copied to a local mailbox (and then perhaps to another) before, typically but not necessarily, being delivered to the Inbox.

The filtering "recipe" - as I will refer to the contents of the .mailfilter file or any other files it includes - also has the capability to pass the message through an external program, which processes the message from stdin to stdout according to parameters in or generated by the filtering program.  In this way, the message itself can be modified.

.mailfilter must have its permissions set only to be accessible to the user, otherwise Maildrop generates an error.  Maildrop compiles the contents of the file and exits with an error if there is a syntax problem with it. An error to this effect will be written in /var/log/maillog .  For instance:

postfix/local[23463]: 3CDCF3D9C: to=<blah@example.org>, relay=local, delay=0, status=deferred (temporary failure. Command output: .mailfilter(33): Syntax error after if )

The (33) refers to the line number where the syntax error occurs.

The intention is that the calling program, the MTA - Postfix in my case - will queue the message for delivery later.  Postfix does do this.  The command:

postfix flush

will cause all queued messages to be delivered immediately.

It haven't figured out how to run Maildrop on the command line to check the .mailfilter syntax, but I am sure it can be done, or achieved by some script.  Without such a test, the way to test that he new filtering recipe does not have syntax errors is to send yourself a test email and check that it comes through.

Setting the filter recipe by web or email?

The filtering language is pretty fancy, and not the sort of thing a non-technical user would want to mess with.  A good project would be a Web or email interface into this filtering capability.  For instance placing an email of a particular format into a particular mailbox via IMAP, where a program would crunch it, convert it to a .mailfilter   file or if there were any errors, annotate the original file with pointers to what needs to be fixed.  Alternatively, Maildrop could have a filter recipe to pass an email with special flags that it contains a new filter recipe to an external program which will extract the recipe, install it, send a test mail, and if it gets through, leave the new recipe in place and send an acknowledgment email.  If the new recipe had syntax errors, then the program would remove it, restore the old one and send something helpful back to the user about whatever errors Maildrop reported (in /var/log/maillog ).  The same system could also make a copy of the filter log available by some means, such as mailing it to the user.

From the same stable as Courier, Courier IMAP and Courier Maildrop, is a cgi-based web-mail system called SqWebMail: http://www.courier-mta.org/sqwebmail/ .  This has a web-based filtering interface for Courier Maildrop, but I haven't checked it out.  

Brief examples of a filtering "recipe"

Here is a simple .mailfilter file:

logfile "maildrop-filter-log"

                         # Search the message body for distinctive SirCam virus/worm text.
                         #
                         # ":b" means search the body.

if(    ( /I send you this file in order/:b)   \
    || ( /Te mando este archivo para que/:b)  \
  )
{
                         # Copy it to a mailbox. Then add a label
                         # deliver it to the Inbox.
                         # The "to" command terminates filtering.
    cc "Maildir/.yyy.xxx"
    DELTAG=1
    xfilter "subjadd [SirCam]"
    to "Maildir"
}
 
 

                          # Now look in the headers only, for a particular line.
                          # Absence of ":b" means search in the headers only.
                          # The "^" means look for this text at the start of the
                          # line only.

if ( /^Delivered-To: postfix-users@cloud9.net/ )
{
    cc "Maildir/.Lists.Postfix"
    DELTAG=1
    xfilter "subjadd [PF]"
    to "Maildir"
}

If statements are supposed to be on a single line - so use trailing backslashes to continue them (or any other statement) over multiple lines.

Be especially careful not to have a blank line after an if (. . . ) line!!!  If the example had been:

if(    ( /I send you this file in order/:b)   \
    || ( /Te mando este archivo para que/:b)  \
  )

{
     .....

then the if statement would have not had any effect and control would fall through to whatever was in the braces.
 
 

Note that the two mailboxes mentioned here are specified by their exact directory name, relative to the user directory.  This is how it works with Courier IMAP, but Maildrop does not rely on Courier IMAP and can deliver to Maildir mailboxes no matter what the IMAP/POP server is:

Maildir/.yyy.xxx/

is a mailbox in the yyy folder (of the INBOX, as are all folders with Courier IMAP) of name xxx

(Note:  See the mb2md page from the parent directory to read more about Courier IMAPD's Maildir mailbox naming and directory structures, and how to convert from Mbox format to Maildir.)

Likewise:

Maildir/.Lists.Postfix

appears to the user as:

Inbox -----------------------
    |
    | --- Lists
    |       |
    |       | --- Postfix ---
    |       |

In the Maildrop' log file, as specified at the start of the .mailfilter file, the two sections above generate results like this when they successfully handle a message:

Date: Thu Aug  2 00:10:52 2001
From: Foo <foo@example.org>
Subj: [SirCam] Fantasies for Mistress.doc
File: Maildir/.yyy.xxx                                                (243295)

Date: Thu Aug  2 00:17:13 2001
From: Foo <foo@example.org>
Subj: [PF] Postfix rocks!
File: Maildir/.Lists.Postfix                                            (2474)

If the filtering recipe specifies sending the message to a mailbox which does not exist as a proper Maildir mailbox, then Maildrop will create an ordinary Mbox format mailbox file of that name instead.

The Maildrop filtering language is rich:

• Extra log file entries can be generated as desired: simply use the "log" command followed by text.
• Files can be conditionally included - which saves long, rarely needed sets of filtering instructions from needing to be compiled except when needed.
• There are text and numeric variables.
• There are if, for and while constructs.
• There are ways of trapping exceptions within a section which would normally be fatal to Maildrop!
• "Although Maildrop evaluates expressions numerically, results of expressions are stored as text literals. When necessary, text literals are converted to numbers, then the results of a mathematical operation is converted back into a text literal."

So I don't see much reason to be concerned about the language being inadequate for any mail filtering operation.

Here are some advanced examples I can think of:

Lets say I get a bunch of emails all the time from Telstra Internet about outages all over the country - but in fact I never look at them because the Subject: lines are all the same, and very few concern me.  A filtering rule could find these messages, and then scan the body to identify those which concern my neck of the woods.  Then, those messages could be given distinctive labels and perhaps copied or delivered to the Inbox in ways which are different from the rest of these messages.

It is possible to use an external programs or scripts to send email to any destination - including to an SMS message gateway.  So the filtering system could alert me via my mobile bone to particular incoming emails.
 

Which headers to look for in mailing lists?

Generally, for mailing lists I find that the "Return-Path: " header # is the most distinctive and stable for all the ways that messages can be sent to a mailing list.  Yahoo Groups lists (and others which run from elzm) are different - the "Return-Path: " is customised for every message.  So the "Reply-To: " seems to be the best header.  But list servers can do all sorts of things!  

Sometimes the "Delivered-To: " header can be distinctive, such as to xxxxx-outgoing@example.org .

Examples

Here is a slightly shortened copy of my original operational .mailfilter file:

my.mailfilter.txt

In fact, as time goes on, it gets more elaborate rules to cope with every possible way some list servers might relay mailing list messages.

Here is a section of the log file it generates:

mailfilter-log.txt 

Update history

• 2003-05-29 Page established, based on August 2001 version.
• 2008-04-29 Updated for current version of Maildrop.  Subjadd slightly edited to improve indentation.