RH7.1 > Postfix > Maildrop > Courier IMAP - How-I-did-it
This documentation has been
superseded. Please refer to the parent directory for Postfix,
Maildrop and Courier IMAP on Redhat 9.0.
. . . and how I gave up trying to install Horde
IMP due to problems with PHP
Late June - early August 2001 Robin Whittle
rw@firstpr.com.au Last major update 11 August 2001.
Minor updates 30 September, 4, 9 & 17 October, 13 November, 9
December 2001. 19 & 23 October 2002 (Update history is at the
end.)
This covers Postfix, Courier Maildrop and Courier
IMAP . In October 2002 I added material here and on another
page regarding how to integrate Spam Assassin and the Anomy
Sanitizer for automatically filtering spam and viruses (malware),
and on how to report on
the message numbers and storage requirements of each mailbox.
This is a verbose documentation - because I find that a lot of problems
are caused by overly terse doco which already assumes you know things
that you don't. Also, I think more people's problems will be
solved if
the description is full and search engines find this page better.
Please let me know of any corrections or improvements to this page.
I received one important correction regarding Postfix within a
day of putting it up.
It is not a proper How-To - it is How I Did It, simplified. It
is not a straight linear thing, but it is a lot straighter than the
journey I followed to find this stuff out!
By the time you are reading it, the latest versions of software will
differ somewhat from what I am describing, but to my knowlege, the same
principles hold. If you find significant differences I should
mention, please let me know!
Back to the web-mail page.
Back to the First Principles page for the world's
longest Sliiiiiiiiiinky, corsetery ads, and all sorts of things
. . .
Intro
This page describes how I installed three sets of software
on a Red Hat 7.1 system. This is not a How-To or a complete
tutorial on setting up mail servers etc. for those who have never done
it before, but it is a more complete documentation than is typical of
IT stuff. Part of the reason for that is that I will forget all
this quite soon (Unix systems admin is not all I do) and will want the
various related things well explained for the next time I come this way
in the years to come.
The three sets of software are:
- Postfix - a fast, clean, secure, easier to handle
replacement for Sendmail.
- Courier Maildrop - to deliver mail received by Postfix
to local Maildir mailboxes and to do mail filtering.
- Courier IMAP - to enable access to incoming mail and the
creation of mailboxes on the server using the IMAP protocol.
Courier is a higher-performance IMAPD than the standard University of
Washington IMAP package.
I was planning on installing the Horde IMP web mail system,
which I had on the server which I am replacing. However I had so
much trouble following the documentation and trying to get the required
library PHP files installed suitable for Horde/IMP (my attempts
resulted in all sorts of versionitis which stopped Apache from running)
that I gave up. IMP is a highly regarded web-mail program under
constant development with a wide user base. So I don't suggest
everyone give up as I do. But I found that for my purposes, I was
happier with a new C++ cgi-bin and daemon web-mail program called
Postman, from the University of Valencia, Spain. See the parent
directory for details. My description of the abortive attempt at
installing IMP is below .
Please see the parent directory for links to
pages which cover how I:
- Installed and improved Postman - a new web-mail program
from the University of Valencia in Spain, fast, powerful and written in
C++.
- Installed Courier SqWebMail - a fast web-mail program
for local (including via NFS) Maildir mailboxes, written in C.
- Installed and improved Courier Maildrop to deliver
messages tagged for deletion.
- Integrated Spam Assassin and Anomy Sanitizer
into the set-up I describe here on this page.
Postfix installation is well documented elsewhere and is
relatively
straightforward - but I discuss it below. Installation of Courier
IMAP
and the required Maildrop program to allow Postfix to write to
Courier's
Maildir format mailboxes is not at all trivial - but with the doco
below,
it should be pretty straightforward.
I also investigated using a recent version of the widely respected
text-based Unix email program mutt . It can access mail
accounts via IMAP, and I did eventually configure it to work, but I had
great difficulty selecting mailboxes and with a few other things.
I don't desperately need it now, and may investigate it later.
Information on mutt with IMAP can be found at: http://mutt.sourceforge.net/imap/
. Also, I found that it couldn't obviously read messages which
are tagged for deletion - which is something I often want to do.
mutt was designed initially for accessing local mailspool and
mailbox files, so IMAP support is not a central part of its
functionality.
My needs here are modest - just a few users accessing the server,
which is connected to the Net via a 56kbps modem. But I do make
heavy use of IMAP email (using Netscape Communicator at present) over
the LAN, and sometimes when I am away via the 56kbps modem.
I do not use web-mail myself, except when I am away from my
home/office. The main users of the web-mail system are several
friends who could want to check their email from many locations and
need to do so via a web browser. I may consider using a text-mode
IMAP compatible mail program such as mutt (via SSH) if I am away.
There are no fancy performance requirements on this system - other
than my desire that the IMAP server provide emails in multi-megabyte
mailboxes very quickly - without having to read the whole file.
The University of Washington IMAPD does it simply by reading the
file. Courier does not keep all the contents of a mailbox in a
single linear "mbox" (AKA "mailbox" format) file (as UW IMAP
does). It uses Qmail's "Maildir" arrangement - several
directories per mailbox, with each email a separate file.
This is faster, eliminates the need for lock files (for multiple
programs accessing a mailbox at the same time) and enables the
mailboxes to be accessed on other machines via NFS, which is not
usually possible with single-file mailboxes due to NFS's inability to
support proper locking arrangements.
Why IMAP?
http://www.imap.org/ is the
University of Washington's IMAP information site, which lists software
which implements the protocol - both clients and servers.
- IMAP is a means of retrieving received emails from a server,
as
is POP3, but it uses a secure challenge and response authentication
protocol, while POP3 sends the password in the clear over the Net.
- IMAP enables the creation of mailboxes on the server. So
the email client need not save sent, received, draft etc. emails on
itself
- but can use the server. (Typically email applications which use IMAP
make it possible to store emails locally and on the server, with easy
moving emails between the two.)
- The server can be told to copy, move, search and delete emails
on and between mailboxes on the server, including the Inbox and all
mailboxes created by the user. This means that if a remote user
logs in and receives a bunch of long emails, only the headers are sent
to the client program
- and the client can tell the server what to delete and where to move
or
copy individual emails. Of course, to read an email, it must be
sent
to the client - though I think the IMAP server is capable of sending
attachments only when required. All this makes a lot of sense for
users who are connected via a dial-up modem, or even worse, through a
flaky, supposedly 9.6 kbps, connection via a GSM or other type of
cell-phone.
- A web mail application is intended to run on a server, and
send pages to a web browser, to enable the remote user to do all the
things they want with email. This means that the user (say, at a
library or Internet cafe) cannot store their email on whatever machine
they are using. Therefore, the mail should be stored on the mail
server (although it could also be stored on the web-mail server, which
is not necessarily the same
as the mail server). So to run a web mail system, I need an IMAP
mail
server. (But I use IMAP anyway.)
I want the resulting system to provide:
- A fast, reliable, secure mail server for several domains and
for relaying mail by local users.
- Fast IMAP access to accounts on this machine, as well as IMAP
mailboxes each user can create.
- Mail filtering to multiple
mailboxes on the server. I want incoming mails to be sent to
various mailboxes for the several dozen mailing lists I am on, but a
copy to remain in the Inbox tagged for deletion next time I do an
"expunge". This will replace my current dependency
on Netscape's filtering, and so means I do not have to always access my
account with a suitably configured copy of Netscape. (Early
August 2001 - I have achieved this and am veryhappy with it.
I can access my mail in any way at all, and the filtering is
a completely separate process. See the separate page from the
parent directory on how I modified Maildrop to do this.)
- A web-mail program for web access to local accounts from
anywhere using a Javascript enabled browser. (Instead of
installing IMP again, I installed Postman. Both can be configured
to work with IMAP servers anywhere in the world, but in this case it is
limited to the IMAP server of the machine it is running on.)
All these will be on the same machine, but in practice the mailboxes
could be on multiple computers accessible via NFS. Postfix and
Maildrop must be on the one machine, IMAPD (the IMAP Daemon) could be
on a separate machine and so could Postman. This would involve
some fancier authentication systems than the standard PAM -> Shadow
Password system I am using. The web browser could, of course, be
anywhere in the world. Other email clients (and web-mail programs
- for instance I also have SqWebMail running) can access the IMAPD and
Postfix.
Maildir format Web-mail email client
mailboxes
/-----------\ /----------\ *********** /---------\ /----------\
| | | | * Inbox * | | | |
| | ====> | Maildrop | ===> * * <======> | Courier | | Postman |
<=== | Postfix | | | =\ *********** | IMAPD | | with | /---------\
===> | MTA | \----------/ | | | IMAP | Apache | | |
SMTP | | | |====> ************ | | <---> | | <-----> | User's |
to & | (Message | filtering | * YYY * | | | | HTTP | web |
from | Transfer | rules in \=>************ * <==> | | | | or | browser |
other | Agent) | .mailfilter * XXX * *** | | IMAP | SMTP | HTTPS | |
MTAs | | <--\ *** * * <======> | | <-\ | Out | | |
\-----------/ \ ************ \---------/ \ \----------/ | |
\ \ V \---------/
\---<----------------------------------------<---------------/
SMTP outgoing mail \ \ /------------------------\
\ \-> | |
\ | IMAP capable Email |
\ | client |
\ | |
\-----------<--------- |< SMTP outgoing mail |
| |
\------------------------/
*** Spam and virus filtering (Added 2002
October 20)
This is where extra programs can go
for filtering viruses ("malware") and spam. Another approach is
to have Postfix detect and refuse to accept spam messages from remote
MTAs (on the left-side of the Postfix box in this diagram). This
would cut down on the volume of mail which needs to be delivered
locally. However this
approach is often unacceptable because some false positives are
inevitable in spam detection and the problems resulting from refusing
to accept non-spam email, without the intended recipient ever knowing
that it has been sent or rejected, cannot be tolerated.. This ***
location is the smtp program of Postfix. Postfix is a collection
of programs, and one way of dealing with malware and spam is to
configure the smtp program to also
use a shell script, which passes each message through one or more
filtering programs, such as:
- Spam Assassin http://spamassassin.org
This is a widely respected spam detection system, which works in
various ways, including by reference to remoted blacklists etc.
- The Anomy Sanitizer http://mailtools.anomy.net Not
actually a virus scanner, in terms of identifying specific viruses
(which would require constant updates to its filtering rules) but a
program to eliminate problems in attachments such as Javascript in HTML
emails, and to rename
or delete executable files and to do other things which generally
render
them less harmful. (This is my rough summary - see the web page -
for
instance it can call a virus scanner.) I have it "defang" various
suspect
HTML elements, and "drop" attached files which are executable.
My approach to running Spam Assassin
and Anomy Sanitizer is to use the xfilter command from within
Maildrop's .mailfilterfile. I have a page on how I did
this, which you can find from the parent directory.
Another
way to run Spam Assassin and Anomy Sanitizer is described by Advosys in
Ottawa:
http://advosys.ca/papers/postfix-filtering.html
.
The Postfix configuration changes required for running the Advosys
script for these two programs are entirely separate from how Postfix is
configured to deliver messages with Maildrop. However, this leads
to them running on outgoing mail as well, not just mail to be delivered
to local users' mailboxes. (Actually, the script becomes part of
Postix's smtp program which
also handles the messages arriving from local clients via the line
entering
the bottom right of the Postfix box above.) Advosys solve this by
running two instances of Postfix, with the first one doing the
virus/spam filtering as usual and the second one responding to a second
IP address purely
for SMTP outgoing messages from local mail clients - therefore
requiring all
the clients be reconfigured. For my own purposes, I think a
better approach
would be to put Anomy Serializer and Spam Assassin in the local
delivery
section of Postfix, just before Maildrop. See my separate page,
from
the parent directory for how I did this - running
them
from within Maildrop. However, Advosys.ca have a lot of
experience with
large mail systems, for companies and educational institutions, and
there
are good reasons why they run two instances of Postfix, and do other
things
to improve efficiency. For instance, their clients may want to
filter
for spam and viruses on all incoming email, but only filter for viruses
on
outgoing mail. This is not because they are spammers, but because
spam
detection is an imperfect science.
Whether Advosys' or my approach is used, the aim is to label each
suspected spam or virus message by adding distinctive headers and/or
distinctive
and human-readable material in the body of the message. Anomy
Sanitizer typically also changes suspect message bodies to "defang"
suspect HTML tags, Javascript, executable files etc. Iin the
standard configuration this has the effect of greatly shortening long
virus messages. Then, it is easy in Maildrop's filtering rules to
decide what to do with suspected spam or malware messages. I use
Maildrop with my "Delivered to Inbox Tagged for Deletion" system, with
Cc:ing to separate mailboxes for suspected spam and malware, with the
addition of a "[SPAM]" or "[VIRUS]" subject header for when they are
delivered tagged for deletion in the Inbox. This way,
all messages are visible to the intended recipient - me - but I don't
have
go manually select and do something with the individual spams and
viruses which are increasingly flooding my Inbox. With Netscape
4.77 and some appropriately named mailboxes which appear in the mailbox
list just under the Inbox, I have become pretty adept at using Cntrl
M, M 0, 2 for spam and Cntrl M, M 0, 3 for viruses.
But with 10 to 20 viruses a day and a similar number of spams -
and with constant growth in these
figures - I am getting sick of it! Mozilla's (and therefore
Netscape
7's) email client has no such numeric shortcuts, so it would be a lot
of
cursor movement all day every day to put this shit somewhere, unless I
can
get some filtering system to deal with most of it.
I think it is best to store these highly suspected spam and
virus emails somewhere, rather than to delete them manually or
automatically as they are first viewed or arrive at the server.
To delete this stuff manually or automatically risks accidentally
deleting a valid email - because I can't be super cautious and 100%
correct in my judgements whenever I do this, which could be any time of
the day. With largely automated detection and copying to spam
and virus mailboxes, supplemented by manual steps for the same purpose
with what gets past the filters, at least I can periodically look into
those mailboxes to ensure I haven't acidentally turfed something good
into these cesspits. This takes the stress out of the manual
filtering actions, which makes the process faster. This manual
selection
and turfing into a mailbox is becoming a necessary step many times when
I
check my email, and so it is a daily and hourly curse, which should be
made
as easy and as possible. In fact, the system I describe on the
other
web page works very nicely. After a few days, I haven't had any
false
positives for spam or viruses, and there have only been a few false
negatives
for spam.
Postfix can be configured to deliver directly to maildir format Inboxes
- see http://www.postfix.org/faq.html#maildir
. Since I want to do some filtering, I want to use Maildrop -
which has extensive filtering abilities.
Mail serving, IMAP and any web-mail system are all potentially
intensive applications. They are commonly used for large systems,
such as for schools and universities. Some special tweaking is
likely to be needed for these systems, such as using separate machines
to run the various servers. I have no such intense
requirements. The mailing lists for Postfix, Courier and IMP are
the best place to find out about high-performance and high-availability
systems. (Postman, at present, has no mailing list or active user
community.)
Installing Postfix
I chose not to compile Postfix from the source, but to use
an RPM which is ready to go. I don't have any fancy
requirements for the very latest version. In several years of
using Postfix, it hasneverto my knowledge put a foot
wrong. Salute Wietse
Venema, who carries on this open-source work at IBM!
The Postfix Web abode is:
http://www.postfix.org
There is a very busy mailing list postfix-users (with several web
archives) and an occasional postfix-announce list too. The
archive I searched was:
http://msgs.SecurePoint.com/postfix/
but watch out for banner ads chewing your bandwidth!
In late June 2001, by looking at one of the download mirror sites,
I could see that the latest stable release (actually the first non-beta
release) is 20010228:
ftp://ftp.planetmirror.com/pub/postfix/index.html
But I went back to the Packages and Port's section of the Postfix site
to find some ready-rolled RPMs. This lead to a site by Simon J
Mudd with RPMs for Intel, Alpha and for System 390 (IBM mainframes):
http://www.ext.ea4els.ampr.org/~sjmudd/postfix/ftp/rh7x/RPMS/i386/
There I chose the package: postfix-20010228_pl03release-4.i386.rpm
09-Jun-2001.
My system was a fresh install of Red Hat 7.1 - on an original AMD
K6 200 with RAID-1 dual 20 gig IBM 7200 RPM drives. (The graphic
installer, which only runs from local CD-ROM, not FTP/NFS/HTTP, enables
the creation of RAID-1 systems without any file tweaking. RAID-1
everything, including /boot and <swap> and the machine keeps
running even if you unplug either of the drives . . which are on
separate cables . . .). (In 2002, I changed
to Red Hat 7.2, with newer 60GXP IBM 7200 RPM drives after all the
original
75GXP DTLA drives died - and I am running a 800 MHz Celeron. I
keep
this machine's software updated via the Red Hat Network https://rhn.redhat.com/ , using the
up2date program, which I think is very good. A single free
account is available to each person, or at least each email address a
person uses . . . )
As part of the RH7.1 installation, Sendmail was installed:
sendmail-8.11.2-14. I installed the relevant security updates to
RH7.1, but none of these affected Sendmail.
For installation and configuration information I consulted the Red
Hat Postfix FAQ and HowTo:
http://www.redhat.com/support/docs/mail.html
The key steps are to stash the old Sendmail stuff somewhere, delete the
Sendmail package, kill the Sendmail processes and then install the
Postfix RPM. As the FAQ says:
mkdir /root/sendmail-old
cp /etc/aliases /root/sendmail-old/
cp /etc/sendmail.cf /root/sendmail-old/
cp /etc/sendmail.cw /root/sendmail-old/
cp /etc/mail/* /root/sendmail-old/
rpm -e sendmail sendmail-doc sendmail-cf
--nodeps
killall sendmail
rpm -Uvh postfi*
This caused an error message, about "myhostname" or "mydomain"
needing to be set in /etc/postfix/main.cf.
Postfix is set up to run via the System V init department (/etc/rc.d/init.d/)and
can be restarted with:
/etc/rc.d/init.d/postfix restart
I tried restarting Postfix several times after setting up/etc/HOSTNAME
and /etc/hosts as I think they should be, but the error
message was repeated. The command "hostname --fdqn"
returned the
full hostname, but still the error message appeared and Postfix would
not
start. For the purposes of this discussion, I will say that my
server
is "zair.firstpr.com.au". (Names in this story have been
changed to protect the innocent. I did all this work on a machine
which was later to become my mailserver. The old one was still
running,
and at changeover time, I changed the host name and IP address to make
the
new machine take its place.)
The problem was fixed when I made the following changes to /etc/postfix/main.cf
:
myhostname = zair.firstpr.com.au
myorigin = firstpr.com.au
mydestination = firstpr.com.au
exp.firstpr.com.au zair.firstpr.com.au
mynetworks = 203.36.57.192/27
The first two solve the problem of Postfix not knowing what host
it
is running on.
The "mydestination" line specified the domains for which
this machine is the primary mail server. I have another server
configured as the secondary, and my DNS set up appropriately with zair.firstpr.com.au
as the primary mail server - the one with the lowest number in the
relevant zone files:
IN MX 10
zair.firstpr.com.au.
IN MX 20
postoffice.telstra.net.
The mynetworks line specifies the subnet of my LAN. Postfix
will relay mail sent from any machine on the subnet, but not from any
machine outside. This is essential. Firstly to enable
it to forward mail from local machines and secondly to stop it being
used to relay SPAM. There are fancier ways of specifying what
machines Postfix will relay mail from - but this is fine for me.
Postfix also has extensive support
for dealing with SPAM (UCE = Unsolicited Commercial Email or UBE), but
I
haven't looked at that yet.
Restarting Postfix is fine now. But I need to set up the
aliases in order that it receive any emails for accounts on zair.
The file /etc/postfix/aliases maps the user names of incoming
email addresses to local account names where they may be
delivered.
As far as I know, if Postfix (or Sendmail) is receiving mail for
multiple domains, then this aliases arrangement makes no distinction
between the same user name of different domains. So xyz@foo.mil
is treated the same as xyz@bar.mil. (These traditional
C example names derive from the WWII US military term FUBAR which is an
acronym for Fucked Up Beyond All Recognition . . . from the same school
which gave us Orl Korrect = OK, I assume.) No doubt there is a
way around this.
Q: How does one set up aliases so that everything
addressed to a domain which is not covered by an existing alias will be
sent to a particular account?
A: See the section below on
Virtual Domains.
Here is an example of some aliases I added:
lucy:
lucy
diamond:
diamond
rw:
firstpr
rwhittle:
firstpr
rwi:
firstpr
The first two are straightforward mappings of an email address to
a
local account. The others map my normal email address and two
alternatives to the account which receives all my mail: firstpr.
I originally wrote the text below in red:
After changing the /etc/postfix/aliases file, it is necessary to run the "newaliases"
program which reads this file and compiles them into a database for
Postfix to use. I created a script in this directory to do this
and to restart Postfix.
#!/bin/bash
# Recompile the aliases file and
restart postfix.
newaliases
/etc/rc.d/init.d/postfix restart
but I stand corrected. Jussi Silvennoinen told me:
I thought you should know that it is not
necessary to restart postfix
after changing aliases-map or
any other map. You can simply run 'postfix
reload' which
refreshes all maps. Postfix will even do it by itself in
time, every n minutes. Postfix
is not sendmail, you dont need to restart
the whole MTA every time you
make a small change. All
configuration-changes can be
applied the same way.
Aliases can be used to run mailing-list programs. I used to have
some incantations for Majordomo - but if I was doing a mailing list
now,
I would use GNU
Mailman .
I tested Postfix, partially at least, by sending an email to an
account on this machine and checking that it arrived in that user's
file at /var/spool/mail/.
Then I checked that the already installed University of Washington
IMAP server was working by accessing that account from Netscape on
another machine. Before I could do this, I had to enable IMAP
reception by xinetd, by the following steps:
- Comment out, with a leading "#", the line "
disable
= yes" in file /etc/xinetd.d/imap.
- Restart xinetd with the command "killall -USR2 xinetd".
(Or "/etc/rc.d/init.d/xinetd restart".)
Update 17 October 2001: Virtual Domains
and
catching email to unknown users.
I don't use these features, but I may in the future. It is
possible to configure Postfix to respond to email for multiple domains,
including handling a user blah@xyz.org differently from blah@abc.org.
Without this, then Postfix handles email to blah in the same way
irrespective of
the domain in the email address, assuming (I think) there is a user
account
on the computer called blah. Also, there is a way of setting up
Postfix to handle any unspecifed user name for each domain in the same
way - a "catch-all" facility which will presumably handle the email in
a constructive fashion, rather than bouce it back with an error message
about an invalid address
etc. I haven't actually done this. I am just pointing to
someone
else's Postfix Virtual Domain Howto:
http://www.cluesoft.be/howto/postfix.htm
Installing Courier IMAP . . . and Maildrop
The above account of Postfix installation was written after
the event, from my notes. This account of installing Courier IMAP
is a "revised stream-of-consciousness" effort. The actual path I
took was more involved than this. The following account retells
the important bits. As far as I know, I haven't left anything
out, but maybe I have.
Courier IMAP also provides a POP3 server but I don't use it.
Courier does not work with the standard linear file user (AKA
"Mbox" or "Mailbox" format) Inboxes: the files typically created for
each user in /var/spool/mail. Instead it needs the
Inboxes to be in
the Maildir format, which is explained here: http://www.qmail.org/man/man5/maildir.html
and http://cr.yp.to/proto/maildir.html
. Initially I thought I would need some jiggerypokery to get Postfix to
deliver incoming mail to such mailboxes. In fact, Postfix can be
configured to deliver to Maildir mailboxes, with a specification for
where they are located in the file system.
However, I want to do some fancy mail filtering, which I
understand
Maildrop is good at, so I am installing Maildrop: http://www.flounder.net/~mrsam/maildrop/
. (Note, to make this easier to read, I am capitalising the
first letter of a number of program names.)
I foresee some drawbacks with this Maildir mailbox scheme:
There's no single file to search if I want to
search
a mailbox by ordinary means. (Of course the IMAP server can search
mailboxes.) To create a single linear file, I could copy the
mailbox contents to a local mailbox file on the computer running the
email client, or use a client which supports multiple IMAP servers at
once (Netscape and no doubt others) to copy entire mailboxes from a
Courier IMAP server to one running UW IMAP.
(Actually, Netscape's local mailboxes are Mbox format, so I can
copy
the contents of a Maildir mailbox on the IMAP server to a local mailbox
and
search them there - especially with an excellent Windows file manager
called
Z-Tree Win, which highlights the searched for text and quickly steps to
the
next occurrence. http://www.ztree.com )
Likewise, migration issues - how to get a bunch of mailboxes
in Mbox linear file format into maildir format. More on this
below - I
wrote a perl script.
Searching a Maildir mailbox is likely to be slower and more
CPU intensive than a linear mailbox, since there could be thousands of
files to read. However, adding, deleting or modifying emails will
be much faster since there is no long file to rewrite. (Both
these turned out
to be the case.)
The multiple file approach must take up more space on disk than
a single linear file (unless you have a fancy file system - I guessthe
Reiser File System would be good for this: http://www.namesys.com as well as
having many other advantages). But see Debian user Daniel Stone
writing to the Postfix Users list on 30 September 2001: For reading
the Maildirs with mutt, XFS is at least twice as fast thanthrough,
far faster than ReiserFS, and bit faster than ext2. All hail XFS! .
When backing up a set of mailboxes, the result is
a
mess of files in an archive, rather than a single file - so only an
IMAP
server could make sense of the results. This is an issue for
archaeologists - including myself - when going back through backup CDRs
in the future looking for particular things. (There's a program
which can create a linear file from a Maildir mailbox directory. . . http://www.qmail.org/qmail-manual-html/man1/maildir2mbox.html
More on this below.)
Unix mail programs, elm at least, don't know about the Maildir
format Inbox. Maybe pine would be better? No. Later
versions mutt can access an IMAP server, but I found it was awkward.
But overall, I want Maildir format mailboxes for performance and so
I can run Maildrop and its mail-filtering facilities. A
comparison
of the standard linear file "Mbox" format and the Maildir format for
mailboxes is at: http://www.courier-mta.org/mbox-vs-maildir/
.
Courier IMAP is GPL open source software from Sam Varshavchik
(Double Precision, Inc.) http://www.inter7.com
. The home page is:
http://www.inter7.com/courierimap/
The home of the Courier Mail Server is: http://www.courier-mta.org/
and this has a complete set of man pages for Courier MTA and its
separately available components, Courier IMAPD and Maildrop.
There are various approaches to user authentication, such as LDAP
and an SQL database (such as MySQL). These are typically required for
large systems, but not in mine. I want to use the standard PAM
(Pluggable Authentication Module) approach used by a standard RH 7.1
system. PAM is a standardised, configurable, interface to various
actual authentication schemes.
In my case, PAM for IMAP (and POP3) will be configured to use the
standard
shadow password system. So the only user names and passwords by
which
IMAP can be used are those of local users.
IMAP over SSL is possible. I would be interested in using
this later. Apparently Courier IMAP works with OpenSSL, and my
system has openssl-0.9.6-3. I will investigate this later.
The doco I read is:
http://www.inter7.com/courierimap/
What I will call the Intro page.
http://www.inter7.com/courierimap/INSTALL.html
Installation instructions.
http://www.inter7.com/courierimap/FAQ.html
FAQ. Includes:
An option I should use to get around a bug which
apparently exists in Netscape Communicator. (I did not use it at
first - I had to recompile afterwards to build it with this option
enabled.)
It needs either the GDBM or Berkeley DB database
library. My installation has GDBM 1.8.0-5. I am not sure
what the package name is for Berkeley DB - but I think I installed
it. (INSTALL.html says that GDBM will be used in preference to
Berkeley DB.)
http://www.inter7.com/courierimap/README.imap.html
Notes on configuring clients, such as Netscape Communicator and
Pine. This is vital, because without proper attention to
configuring the client, all sorts of things will go wrong.
http://www.inter7.com/courierimap/BUGS.txt
Bugs in the way certain user agents behave - such as flaky behaviour by
Netscape Communicator when creating a folder and immediately accessing
it, or loop endlessly on attachments with backslashes in the filename .
. .
StarOffice and some versions of Outlook Express being non-standards
compliant
. . . .
There is also a Courier-users list with
public archives at: http://lists.sourceforge.net/lists/listinfo/courier-users
. Just as searching the Postfix mailing list for "Courier" should be
useful, so should be searching the Courier list for "Postfix Maildrop"
. . but I didn't find anything on how to get them working
together. As I did
this installation, some helpful messages were posted, and I posted some
questions and answers too.
The file I downloaded was at:
http://www.courier-mta.org/download.php#imap
courier-imap-1.3.8.2.tar.gz (13-Jun-2001, 1.1M)
Do not un-tar.gz it as root - you must be a mortal user to compile and
install Courier IMAP.
There is a file named 00README.NOW.OR.SUFFER which urges me not to
do a quickie job of this . . .
The first thing to do is remove the existing University of
Washington IMAP package: imap-2000-9. As root:
rpm -e imap
Squirglinax!
Courier IMAP can be installed using normal scripts for compilation
and installation, or it can be made into an RPM. At first, I
didn't understand, but now I think the idea is that by making an RPM,
and then installing the RPM, this installation can be removed with rpm
as well. Salute those who created RPM!
However, the RPM approach, as described in the doco, does not
allow
for any changes to the options which can be built into the program(s).
Later, I had to change one of these, so I had to do a compilation quite
separate from this initial RPM installation. The Intro file (just before the "Documentation"
heading) describes a way of changing the options for an RPM build, but
from what I can see, this only changes how the package is built, but
not the things we want to change as per the INSTALL.html file.
The 15 page INSTALL.html file does not mention anything about
RPMs. It has all the things to configure before compilation, and
then how to compile and install. The first part is done as a
mortal and the second as root.
The RPM instructions are in the "Intro" page: http://www.inter7.com/courierimap/
. This says it is necessary to read all the INSTALL.html
stuff as it "contains many important details you need to know".
But as far as I can tell, with the RPM build procedure, there is no way
of changing options such as the --enable-workarounds-for-imap-client-bugs
option.
The Intro page also points to a section in FAQ.html on making the
RPM as a non-root user, which is the way it must be done. (This
section implores me to read the "Maximum RPM" book, but I will pass on
this.) This section of FAQ.html tells me how I have to muck around with
the configuration of the RPM system before I can do this. See
below: Making the RPM.
So it seems that I need to:
- "Disable" any xinetd entries for imap and pop3
. (Courier IMAP does not use xinetd.) This is
already done. Removing the UW IMAP package also removed these files in /etc/xinetd/
.
- Read INSTALL.html to see if there is anything I need to change
about how Courier IMAP will be built.
- If there are no changes, make the RPM as described in "Intro".
- If there are changes, follow the procedure in the "Intro"
page's instructions on extracting and changing "courier-imap.spec" and
using that to control the RPM creation process.
So I read over INSTALL.html . . . (Note, I didn't change any of these
for my initial RPM build - I can't change them for this method of
building
Courier IMAP as far as I know.)
I am not using LDAP or MySQL - just ordinary PAM user
authentication. I have MySQL 3.23.36-1 installed (I needed it for
IMP - at this stage I was still planning on installing IMP) but I am
not planning on using it
for IMAP. The RPM generation process will apparently create up to
three RPMs, or two in my case since I don't have LDAP. I will get
an ordinary RPM and another one suitable for use with MySQL (or was it
an
extra package to add MySQL functionality?).
There is a worrying looking section on how I will have to tell
the PAM library how to authenticate the "imap" service . . "What you
need to tell your PAM library is something that you will have to figure
out by yourself, because it depends on the version of your PAM library,
and your operating system." This sounds like the sort of headache
I could do without - perhaps the cause of authentication problems with
Courier (or was it Cyrus?) IMAP and Red Hat in the past (cross
fingers). This is something to sort out after it is compiled and
installed - or in my case compiled, RPM-packaged and RPM
installed. There is guidance - I will sort it out later.
(Actually, the way the package was built was fine in respect of PAM.)
Another post-installation thing is that I will have to ensure
that Courier IMAP runs when the system boots up.
The package I built installs the appropriate file /etc/rc.d/courier-imap
.
There is some configuration to do after installation to make
IMAP-SSL work with openSSL. Likewise some start and shutdown
scripts for this too. I will leave SSL out of this for the
time being.
There are similar config files and start / stop scripts for the
POP3 daemon and its SSL sibling. I won't bother with this.
Options to configure: The ones which look
relevant to me are . . . only one:
--enable-workarounds-for-imap-client-bugs
- there are a number of various bugs in certain IMAP clients. The
current list of broken IMAP clients consists of Netscape Messenger and
Sun's StarOffice. This option enables some workarounds for some bugs in
these clients, however, note that this may break compatibility with
software that correctly implements IMAP4rev1. Additionally, "make
check" will fail when this option is used. See imap/BUGS.(html|txt)
for more information. NOTE - if this option is used, make check
WILL FAIL. You should first configure Courier-IMAP without this option,
run make check, then reconfigure Courier-IMAP with this
option.
Initially, I built Courier IMAP without this option.
This text describes the various installation directories - in /usr/lib/courier-imap/.
I don't need to change them.
There are instructions on adding these directories to the
system's $PATH and $MANPATH.
There is a lengthy section on authentication modules.
RH7.1
uses shadow passwords - and PAM. So Courier IMAP could be
configured
in two ways:
- To use just the shadow password system: the authshadow
module.
(This is what I ran eventually, at least with Red Hat 7.2.)
- To use the authpam module, (PAM in the OS uses
shadow passwords, on my system, as far as I know). This entails
configuring the RH7.1 PAM system to work with Courier's authpam.
We are encouraged to use authpam, but if we can't figure it out, we can
"throw in the towel" and revert to authshadow, which requires no
configuration. Apparently the pre-compilation script for
configuring Courier will figure out which approach to use, and I expect
it will select authpam on my system. In fact, the authentication
was fine, but I suspected it was crook when I got problems which in
fact were caused by incorrect mailbox locations.
Other than my concern about --enable-workarounds-for-imap-client-bugs
there seemed to be nothing I need to change before
compiling-RPMing Courier IMAP. So I proceeded with the
instructions given in the "Intro" page.
Making the RPM
From "Intro":
Just as the source cannot be
compiled by root, the Courier-IMAP RPMs cannot be built as root either.
However, by default, RPM is not configured to be used for building
software as a non-root user. It will be necessary to make some minor
changes to the RPM configuration to enable building as a non-root user.
From FAQ.html:
First, you need to create a mirror
image of the main RPM directory in your account:
mkdir $HOME/rpm
mkdir $HOME/rpm/SOURCES
mkdir $HOME/rpm/SPECS
mkdir $HOME/rpm/BUILD
mkdir $HOME/rpm/SRPMS
mkdir $HOME/rpm/RPMS
mkdir $HOME/rpm/RPMS/i386
Use sparc, or alpha, or
whatever's appropriate. Finally:
echo "%_topdir $HOME/rpm" >> $HOME/.rpmmacros
That's it, now you can build your RPMs:
rpm -ta
courier-imap-0.34.tar.gz
I use Midnight Commander rather than this
stuff. I log in as user robin
and create the directories in /home/robin/
. Then
I create a file there called ".rpmmacros"
which contains the text:
%_topdir
/home/robin/rpm
(One person who wrote to me forgot to do this and had all sorts of
trouble.)
I have the file courier-imap-1.3.8.2.tar.gz in /home/robin/
. Whilst invoking the favours owed to me by all relevant deities, I
type:
rpm -ta
courier-imap-1.3.8.2.tar.gz
and . . .
. . . my machine was in the grip of a higher intelligence . . .
9 minutes later, it was a whole lot smarter.
It created in my rpm directory:
SRPMS/courier-imap-1.3.8.2-1.7.1.src.rpm
RPMS/i386/courier-imap-1.3.8.2-1.7.1.i386.rpm
RPMS/i386/courier-imap-ldap-1.3.8.2-1.7.1.i386.rpm
RPMS/i386/courier-imap-mysql-1.3.8.2-1.7.1.i386.rpm
So I must have LDAP installed after all. The LDAP and MySQL RPMs
are small - just 27 k or so - compared to the 280k main binary RPM.
I tried installed that main binary RPM, but it failed. I
need
to be root to do this:
rpm -i courier-imap-1*
There were no errors.
The installation puts a bunch of directories at /usr/lib/courier-imap/,
including a bin/ directory which contains the imapd
executable. Once installed, the command:
rpm -ql courier-imap
lists where all its files are installed.
See below for how I recompiled
Courier IMAP to work with Netscape's bugs. I think I figured out how to select options when
building
the RPM. The following is for my initial installation.
The INSTALL.html doco has instructions on starting and stopping
Courier IMAPD at bootup and shutdown, but I found that an entry for it
had been automagically added to the System V init directory:
/etc/rc.d/init.d/
This proved to work fine.
I have a script which reports on the state of these things:
chkconfig --list > /root/init.d.txt
I restarted Courier IMAPD:
/etc/rc.d/init.d/courier-imap restart
PAM authentication for Courier IMAP
(Note 2003 June 1, when I am compiling Courier IMAP 1.7.3 on this
machine
which is now Red Hat 7.2, I find that in recent times the following two
files
were not in the system, so the following doco may be
misleading. With
the RH 7.2, I changed authdaemonrc to include only authshadow - so at
first
I used pam, with RH 7.1 and with RH 7.2 I used just authshadow.)
By the magic of the RPM two similar files have been installed for PAM:
/etc/pam.d/imap
/etc/pam.d/pop3
The imap one contains:
#%PAM-1.0
#
# $Id: system-auth.authpam,v 1.1
2001/02/02 05:42:57 mrsam Exp $
#
# Copyright 1998-2001 Double Precision,
Inc. See COPYING for
# distribution information.
#
# This is a sample authpam configuration
file that uses pam_stack
# (circa linux-pam 0.72).
auth
required pam_nologin.so
auth
required pam_stack.so service=system-auth
account
required pam_stack.so service=system-auth
session
required pam_stack.so service=system-auth
I have never done anything with PAM before. This file looks
similar to the ppp one (which I have not configured or tested
yet) but different from the rest. All the rest, including those known
to work, have /lib/security/ in front of the pam_stack.so
and other pam_*.so file
names. So I changed this file to:
auth
required /lib/security/pam_nologin.so
auth
required /lib/security/pam_stack.so
service=system-auth
account
required /lib/security/pam_stack.so
service=system-auth
session
required /lib/security/pam_stack.so
service=system-auth
This probably makes no difference.
I had some trouble initially, which was later found to be to do
with mailbox locations. I thought it was problems with
authentication, so I changed a few things which perhaps I didn't need
to alter.
I spent nearly an hour reading the PAM doco in /user/share/doc/pam-0.74/html/
starting with pam.html. However there was no
mention there of the "pam_stack.so" module - which is used in
the imap and most of the other PAM config files. A
search of all the HTML files reveals no mention of "pam_stack".
Here is a bit of an explanation of what is happening with this PAM
stuff.
There is a man page for pam_stack! pam_stack.so service=system-auth means use
the stack of PAM libraries etc. defined in /etc/pam.d/system-auth.
The pam_nologin line simply prevents non-root users if
there is a file called /etc/nologin. But there is no
such file on my machine.
The next line is identical to that which most other services rely
on, such as FTP, which work like a charm.
Courier IMAP's INSTALL.html has extensive details on various forms
of authentication. My RH7.1 system uses shadow passwords - and
this is all I need because I am only doing IMAP for users of this
system - so in theory, all I need is the authshadow module.
This does not go through PAM at all. The doco encourages me to
use PAM, where the control of authentication is specified in the
/etc/pam.d/imap file - and which would ultimately come back to using
the shadow password system. As the doco says:
Additionally: PAM is used only for password validation.
The home directory and the user and group IDs for the account are still
taken from the system password file.
Perusing the config files at: /usr/lib/courier-imap/etc/ I
find that in the main configuration file /usr/lib/courier-imap/etc/imapd
the authentication is handled by a separate program: authdaemon.
The config file for this, in the same directory: authdaemonrc
lists all the authentication alternatives as being active:
authmodulelist="authcustom authcram authuserdb
authldap authmysql authpam"
These are all described in the INSTALL.html file. (This was when
I was having trouble which later turned out to be due to mailbox
locations.) I could get rid of the lot and put just authshadow
there, but I will remove all but authpam. The
RPM-magic-compilation auto configuration script has added the mySQL one
because I happen to have mySQL installed, and I am not sure about the
others - but I only want the PAM one. So I altered this line to
be:
authmodulelist="authpam"
and restarted Courier IMAPD:
/etc/rc.d/init.d/courier-imap restart
Later, I changed it back to the original state.
Recompiling Courier IMAP with the
Netscape bug workaround option
I did this later, but will describe it here. After I did this,
I learnt what is probably the way to build an RPM with various options.
See below.
Refering to the Intro file,
just above the "Documentation" heading, it looks like there is a way of
controlling the options to the way the RPM is built.
Copy courier-imap-1.3.8.2.tar.gz
to your %{_topdir}/SOURCES directory, extract the file courier-imap.spec,
then copy it to your %{_topdir}/SPECS directory. Make your
custom changes to the spec file, then run "rpm -bb
courier-imap.spec".
But as far as I can see this only affects the way the RPM package is
built, not the options for how the Courier IMAP binaries are compiled.
So I did a completely different approach to compiling Courier
IMAP. I worked from the INSTALLATION section of INSTALL.html.
In a completely different directory from my RPM build, as user robin
, I unpacked the tar.gz file to create the directory
/opt/courier-imap-1.3.8.2/
Now, the first step in the compilation is to run the configure script
with any required options. The one I want is for --enable-workarounds-for-imap-client-bugs
From this directory, as user robin, I gave the command:
./configure --enable-workarounds-for-imap-client-bugs
| Now I see how options are
selected for compilation, I can see how they could be used for an RPM
build.
The instructions for building an RPM with options are in the
Intro file - but it is not helpful
about how to
set options. Now I understand that these options, such as the one
in
red above, are options to the main configure script, I can see that if
you
follow the instructions and edit the lines in courier-imap.specfile
which begins with:
%configure . . .
to add options such as the one in red above, then the RPM build process
will use these options.
My difficulty in learning this from the documentation is a
typical problem which arises when a project grows and changes, and the
author doesn't fully recognise what important things are not obvious to
a newbie reading the doco for the first time.
|
Then I gave the command:
make
This creates some binaries, including the imapd binary
in directory imap/.
Then:
make check
As the doco notes:
# Note - the --enable-workarounds-for-imap-client-bugs
# option to configure will result in make check FAILING.
There was a bunch of errors.
Then:
su root
At this point, I didn't want to install any programs, all I wanted
was the new imapd binary. I don't want to upset my
current installation. I assume that only the imapd binary is
affected by this bug workaround option - so I just want to get the new
binary and put it in place of /usr/lib/courier-imap/bin/imapd
So I skipped the rest of the instructions.
The compilation produces binaries full of debugging labels - it is
561,298 bytes.
To get rid of these:
strip imapd
This reduces it to 162,992 bytes.
Then I copied it to /usr/lib/courier-imap/bin/imapd and
made it user/group root and permissions 775 like the one I was
replacing.
Then I restarted it with:
/etc/rc.d/init.d/courier-imap restart
Then, Netscape 4.77 was able to read the contents of the Inbox. without
any trouble. I was also able to create subfolders correctly, but
there were some confusing things to watch out for. Selecting the
Inbox and then right trackball-clicking it to create a new sub-folder
would (on Linux) try to create it under "zair.firstpr.com.au" rather
than under "Inbox" -
so be sure check and manually select the proper location from the
pulldown
list. With neither Netscape or Outlook Express could I create
mailboxes
except as submailboxes of the Inbox. This is how Courier IMAP
works. More on this below.
Update 9 October 2001:
Mozilla has trouble opening IMAP mailboxes. After
many frustrations with Mozilla hanging, I read on the Courier users
maling list on 8-9 October that there is a majik inkantation to fix
this. This was changing a setting in "etc/imapd" and I
thought "Urggh? What file /etc/imapd??". But this is a
shorthand for the easy
to forget config file, one of several in this directory:
/usr/lib/courier-imap/etc/imapd
Phil Brutsche wrote, in part:
I had problems with
Mozilla/Netscape 6.x hanging as well until I set the
MAXPERIP variable in
etc/imapd to be something larger than 5. I also
increased the MAXDAEMONS
variable to be something larger than 10 (I
currently have MAXPERIP=10
and MAXDAEMONS=40).
If you have more than 1 or 2
people using Mozilla/Netscape 6.x with Courier
IMAP I would set MAXPERIP and
MAXDAEMONS to be very large - 1000 would be a
good number to start with
(that allows for 200 people to use
Mozilla/Netscape 6.x).
Keep in mind that I don't
know what affect making that change will have on
Courier IMAP - I don't have
very many people accessing my mail server, as
you can guess :)
I had:
MAXDAEMONS=40
MAXPERIP=4
Since serveral users could be using the IMAP server from the same
computer, via a web-mail system, I decided to set these higher.
Also, I can do some pretty intense hammering of IMAP mailboxes
all at once from the one
email client. For instance, I could be searching opening, reading
etc.
MAXDAEMONS=100
MAXPERIP=20
Mozilla was much happier!
In October 2002, whilst reviewing this page, I also found that I had
made two other changes to this config file. The first one, if I
remember correctly, makes no real difference - the check all folders
command is either not used on Mozilla, or is not used correctly, or
maybe its not a very useful or well enough defined command in the IMAP
protocol to be really useful.
IMAP_CHECK_ALL_FOLDERS=1
The second one is to do with starting up the IMAPD deamon:
IMAPDSTART=YES
. . . Maildrop
The Maildrop doco I read is:
(30 September 2001: I was reminded by a reader that some things can be
set in the /etc/maildroprc file. Please see the Maildrop man page
for
this:
maildrop-man.txt .)
There is also some doco on mail filtering, but at this stage in the
installation process, I was not interested in this.
Please refer to the parent directory for a page with
three things relevant to Maildrop:
- My changes to the source code to support delivery to mailboxes
with the message flagged for deletion. (My DELTAG mods.)
- A separate program to add labels to the Subject: lines of
messages - subjadd.
- Some examples of how I do filtering - the standard doco is
terse and has virtually no examples.
I downloaded Maildrop from: http://www.courier-mta.org/download.php#maildrop
a file maildrop-1.3.3.tar.gz 07-Jun-2001 565k.
(Update 2002 October 19: I downloaded the source for a
development snapshot of version 1.4.0
maildrop-1.4.0.20021014.RC2.tar.bz2 and am now using this
version, but not with the following method. Please see my
Maildrop modifications page from the parent directory for more
details..)
I read all the doco and didn't see any configuration items I
needed
to change, so I built the RPM. The doco says nothing about not
being
root. As root, in an otherwise empty directory, I tried:
rpm -ta maildrop-1.3.3.tar.gz
Sure enough, my machine was again illuminated by something
from on high . . . .
Five minutes later . . . what happened? No matter where I
gave the above command, the results were in:
/usr/src/redhat/RPMS/maildrop-1.3.3-1.7.1.rpm
259,239
/usr/src/redhat/RPMS/maildrop-devel-1.3.3-1.7.1.rpm
58,400
/usr/src/redhat/RPMS/maildrop-man-1.3.3-1.7.1.rpm
58,763
I installed the first package:
rpm -i maildrop-1*.rpm
I think this "Compile to RPM and then install it" approach is excellent
!
I also installed the maildrop-man package, but not the
maildrop-devel - which is C libraries which I don't think I need.
Once installed, the command:
rpm -ql maildrop > maildrop-files.txt
lists where all its files are installed.
Sidebar: For my own reference at
least, here are some RPM commands of
general usefulness:
rpm -qa > rpmlot.txt
Lists
the names of all installed packages, I assume in order of their
installation.
rpm -qa | sort > rpmlot.txt
Ditto
alphabetically sorted.
rpm -qal > rpmlotfiles.txt Lists
all the files of all installed packages. There is no clear indication
of what package the files belong to - but it is not too hard to figure
it out.
rpm -qf <file name> Lists the
package a file belongs to.
|
The INSTALL.html file shows what files are installed and where - but
this does not match what happens in my system. For instance,
according to INSTALL.html, the binaries are in /usr/local/bin,
but in fact they are installed in /usr/local
The executable is at:
/usr/bin/maildrop
Now I need to figure out how to make Postfix use Maildrop . . .
Get Postfix to use Maildrop
As noted previously Postfix is capable of delivering mail
directly to Maildir mailboxes. See: http://www.postfix.org/faq.html#maildir
- however later, I want to use the filtering capabilities of Courier
Maildrop. I could not find specific instructions in any doco on
how to configure Postfix to use Maildrop, but I found a few things in
the archives of the Postfix mailing list. According to one
message (http://msgs.SecurePoint.com/cgi-bin/get/postfix0102/395/1.html)
it is simply putting "maildrop" into either the mailbox_command or
mailbox_transport parameter in /etc/postfix/main.cf.
Here is the relevant section of that file:
# The mailbox_command parameter
specifies
the optional external
# command to use instead of mailbox
delivery. The command is run as
# the recipient with proper HOME, SHELL and
LOGNAME environment settings.
# Exception: delivery for root is
done as $default_user.
#
# Other environment variables of interest:
USER (recipient username),
# EXTENSION (address extension), DOMAIN
(domain part of address),
# and LOCAL (the address localpart).
#
# Unlike other Postfix configuration
parameters, the mailbox_command
# parameter is not subjected to $parameter
substitutions. This is to
# make it easier to specify shell syntax
(see example below).
#
# Avoid shell meta characters because they
will force Postfix to run
# an expensive shell process. Procmail
alone is expensive enough.
#
# IF YOU USE THIS TO DELIVER MAIL
SYSTEM-WIDE, YOU MUST SET UP AN
# ALIAS THAT FORWARDS MAIL FOR ROOT TO A
REAL USER.
#
#mailbox_command = /some/where/procmail
#mailbox_command = /some/where/procmail -a
"$EXTENSION"
# The mailbox_transport specifies the
optional transport in master.cf
# to use after processing aliases and
.forward files. This parameter
# has precedence over the mailbox_command,
fallback_transport and
# luser_relay parameters.
#
# Specify a string of the form
transport:nexthop, where transport is
# the name of a mail delivery transport
defined in master.cf. The
# :nexthop part is optional. For more
details see the sample transport
# configuration file.
#
#mailbox_transport = lmtp:unix:/file/name
#mailbox_transport = cyrus
So it seems that
mailbox_command =
/usr/bin/maildrop
will do the trick. (Restart Postfix after changing the
config
file!)
Also in the main.cf file I made this change:
local_destination_concurrency_limit=1
Because Maildrop can only deliver one email at a time and so
should
not be asked to deliver to two or more local mailboxes at the same
time.
I haven't used mailbox_transport or fallback_transport.
I don't understand enough about Postfix to know whether I should I set
these to point to Maildrop too. I suspect they don't affect me in
my simple setup of delivering to local users. There are all sorts
of complex delivery options, via various means, to files on different
machines, which are discussed on the Postfix mailing list - but I don't
understand them.
But when I first made these changes to Postfix's main.cf
file, Maildrop ran (as evidenced by lines in /var/log/maillog),
but mail received by Postfix for user blah was still
getting written to an already existing ordinary Mbox format single file
Inbox at /var/spool/mail/blah . If I deleted the Inbox
file, it is regenerated. It seems that Maildrop will write to an
ordinary /var/spool/mail file if it does not find an already
existing Maildir file where it expects it.
It turns out that Maildrop needs to be recompiled to change
where it delivers its mail to. Changing Postfix's main.cf
does not affect Maildrop's behaviour. In order to get it
working, I also needed to create a proper Maildir format Inbox.
The following sections deal with these two matters:
- Recompiling Maildrop to put mails where I want them - or
rather
where Courier IMAPD expects them.
- Location of the Inbox and other mailboxes and some thoughts on
converting from old Mbox format to the new Maildir format.
Recompiling Maildrop to put mails where I want them
As described below, Maildrop needs to be configured to put mails for
user blah in a Maildir format directory called:
/home/blah/Maildir/
Scrutinising the "DEFAULT_DEF" section of:
http://www.flounder.net/~mrsam/maildrop/INSTALL.html
I find that I should set this DEFAULT_DEF to "./Mailbox
" to make Maildrop deliver incoming mail as Qmail would, and as Courier
IMAP
expects, to /home/blah/Maildir/ .
Now I want to recompile just the maildir binary, without
mucking around with RPM.
In the unzipped source tree, in /maildrop-1.3.3/ , as
root, I give the command:
configure
Now I want to alter this DEFAULT_DEF thang . . .
It is mentioned but commented out in the /maildrop-1.3.3/maildrop/config.h
file which the configuration script has generated. (Or was it
uncommented and set to /var/spool/mail? I am unsure.
This report is a simplification of the path I took.)
I uncomment it and make it:
#define DEFAULT_DEF "./Maildir"
Then I change back to the /maildrop-1.3.3/ directory and
type:
make
After it is all done, I have a new binary in the maildrop
directory
"maildrop". But it is 604,043 bytes, while the old one
was
149,864.
This is mainly debugging symbols, so:
strip maildrop
reduces it to 136,136 bytes.
I rename the old maildrop binary, copy the new one to /usr/bin/
and give it the right permissions. What were those permissions?
In October 2002, I am not sure, but it has been running fine with
owner root, group root (maybe should be mail) and permissions 10755.
Success! The new executable delivers mail to /home/blah/Maildir
. (Actually to the /new/ directory there.) Swapping
back to the old binary, the incoming mail goes to /var/spool/blah/
again. Since Maildrop is called by Postfix, there is nothing to
restart.
Location of the Inbox and other mailboxes
I have found that Courier IMAP, as I installed it on my RH7.1 system
(by using rpm to compile it to an RPM package, and then installing the
package without any changes) has the following arrangements for user blah
:
/home/blah/Maildir/
This needs to be created by some means (say by the useraddbinary
and a suitable prototype directory in the /etc/skel/
system). At my current stage of testing , I use Courier's maildirmake
binary:
http://www.flounder.net/~mrsam/maildrop/maildirmake.html
as the user blah, to make this directory. From /home/blah/
simply:
maildirmake Maildir
This /home/blah/Maildir/ directory is a Maildir format
directory. Its owner is blah and it contains three subdirectories
which, like itself, are chmod 700.
/home/blah/Maildir/cur
/home/blah/Maildir/new
/home/blah/Maildir/tmp
Together, these directories constitute a Maildir format "Inbox" which
I can access with both Netscape 4.77 and Outlook Express.
Furthermore, using the client (Netscape or Outlook Express) I can
create mailboxes within this directory, and then mailboxes in those.
(Netscape is configured as per: http://www.inter7.com/courierimap/README.imap.html
)
Here is an example of user-created mailboxes. Each of these
of these mailboxes is a Maildir directory with its three
sub-directories.
To the client they appear as a tree of directories and
sub-directories, such as:
Inbox
|
|--aaa
|
|--Trash
|
|--xxx
|
|--yyy
|
zzz
Physically in the file system, they are as follows, not counting
each directory's three subdirectories and two or three files which
Courier IMAP puts
in each one.
/home/blah/Maildir/.aaa
/home/blah/Maildir/.Trash
/home/blah/Maildir/.yyy
/home/blah/Maildir/.yyy.zzz
Each can contain both messages and mailboxes.
I have not figured out the concept of shared mailboxes. This
is an advanced approach suitable for having several users access common
mailboxes, with the ability to mix their mails in those mailboxes and
only being able to delete their own emails. I am not going
to pursue this.
So with Courier IMAPD, from the client's point of view, all
mailboxes are sub-mailboxes of what appears as "Inbox".
This is actually a bit of a pest. It is an extra keystroke
when using Netscape's Alt M, M command to move a message, since I must
press 0 to select the Inbox before I can start selecting which folder
and mailbox I want to put it in. However it has the advantage
that a user's entire mailbox system, including their Inbox, is all at /home/blah/Maildir/.
In the file system, all mailboxes are direct subdirectories in /home/blah/Maildir/
and their actual position in the tree of mailboxes seen by the client
is encoded in their names. So ".yyy.zzz" is really mailbox "zzz"
which is appears inside "yyy" from the point of view of the IMAP user.
With my previous University of Washington IMAP server, the
arrangement was that all mailboxes were simple linear files.
The Inbox for user blah was:
/var/spool/mail/blah
The user-generated mailboxes, at least the
way I set them up, using the same example mailboxes as above, are at:
/home/blah/mail/aaa
/home/blah/mail/Trash
/home/blah/mail/yyy/
/home/blah/mail/yyy/zzz
In this arrangement, "yyy" is a folder for containing mailboxes, it is
not a mailbox itself.
Maildrop filtering and my two
extensions to its capabilities
I have added two features to Maildrop's filtering
abilities:
- The ability to deliver an email so that it is tagged for
deletion. (Source code changes to Maildrop.)
- The ability to add a header, such as [Postfix] to the subject
line of an email. (External filter program called subjadd .)
The full details and doco on these is in a separate directory.
This also discusses my exploration of Maildrop filtering.
>>>../ Maildrop-mods-filtering
/ <<<
Here is a brief discussion of why I want the two additional features.
The ability to add a label to the subject line is useful,
for reasons including:
- Labelling all emails from a mailing list, or some other
source,
for easy visibility and for alphabetic sorting in the Inbox.
- Potentially the labelling of SPAM, if the Maildrop filtering
scheme, or any external programs it invokes, can figure out what is
SPAM and what is not.
Here is a short version of why I want to be able to put messages in the Inbox tagged for deletion.
Please see the above-mentioned directory for a full discussion of why I
want to do this and move mail sorting onto the server and so not depend
on the client.
I want to replicate the functionality of Netscape's mail
filtering system - but to do it entirely at the server. I am on
several dozen mailing lists and so filtering is vital. I have
Netscape set up to copy messages to the particular mailbox for that
mailing list, and to leave the message in the Inbox, but flagged for
deletion. The filtering happens when I "Alt-F M" Netscape
to check for new mail. The headers come to Netscape and it sends
commands back to the IMAP server about where to put the messages and
whether to set the "T" tagged for deletion flag for messages which also
remain in the Inbox.
The great benefit of this is that I can see all the activity on
my lists without having to look into their various mailboxes.
When I have scanned the "tagged for deletion" messages, (which I can
read, print, reply to or untag with the Del key) then I use "Alt-F T"
to expunge the mailbox. Netscape tells the IMAP server to delete
them all.
(Later . . . I have set this up and it is an excellent
thing!)
Converting old Mbox format mailboxes to Maildir directories
I wrote a Perl script which solved my
problems of converting old Mbox mailboxes to the new Maildir
format. This is "mb2md" and it has its own Web-abode:
>>> mb2md
<<<
The exact details of how I converted my 150 or so old mailboxes are not
important. But basically it was this:
- Make sure the mailboxes and the directories there were in had
no spaces or dots in their names. (I found it more convenient to
rename them via Netscape's Messenger program, rather than actually
changing file names in the server, so I could just edit the name,
rather than having to re-type it with Unix commands, or Midnight
Commander, which is what I normally use.)
- I took this opportunity to clean up the names of mailboxes and
directories so I would be happy with them for a while. I made
every
mailbox start with an upper-case letter. I used no spaces or
underscores
etc. I only used hyphens to separate words to form a single
mailbox
name.
- A shell script to make multiple calls of mb2md for:
- The /var/spool/mail/blah mailspool.
- Some mailboxes I want to appear in the root directory from
IMAP's point of view. With Courier IMAP, this lowest level is within
the Inbox, whereas with UW IMAP, the Inbox and other mailboxes were all
at the same lowest level.
- For each of the directories of mailboxes - "aaa" and "lists"
in the example below.
I didn't do this on a live machine. I disconnected the old server
from the Net, copied all the relevant files via NFS to the new machine,
did the conversions, took the old machine off the LAN, reconfigured the
new machine to take the place (name, IP address etc.) of the old one,
and
then reconnected to the Net. Because it was basically my large number
of
mailboxes, and a few mailboxes for a few other users, I did not
implement
any master plan to convert multiple user's mailboxes automatically.
If you develop such a scheme, let me know, because other people reading
this
page would no-doubt like to use it.
In my old arrangement, I had some mailbox files in my main
directory ~/mail/ (Sent, Drafts and a few others) and the rest in a
single level of sub-directories of this. For example
/home/blah/mail/Sent
/home/blah/mail/Drafts
/home/blah/mail/aaa/xxx
/home/blah/mail/aaa/yyy
/home/blah/mail/lists/analogue-heaven
/home/blah/mail/lists/akai-sampler-list
I want to keep this overall arrangement, but rename a few of
them. The mailbox files were generally made via Netscape, and I
foolishly made some of their names contain spaces. I want to turn
the above into
Maildir mailbox directories something like:
/home/blah/Maildir/Sent/
/home/blah/Maildir/Drafts/
/home/blah/Maildir/.aaa.xxx/
/home/blah/Maildir/.aaa.yyy/
/home/blah/Maildir/.lists.analogue-heaven/
/home/blah/Maildir/.lists.akai-sampler-list/
I wrote mb2md (see above) to
solve these problems. It can convert an entire directory of Mbox
mailboxes in one operation, for instance, everything in /home/blah/lists/
. The mb2md page also has patched
versions to do things my script did not do, such as make the message
file dates follow that in the "From " line of each message in the
origional mbox - which keeps some clients such as Outlook Express a lot
happier.
Below, is an exploration of some pre-existing alternatives, and a
program which turns a Maildir into an Mbox mailbox file.
Probably the best single source of Maildir conversion utilities
is to be found in the User Contributed Maildir Support section of
the qmail home page, which is on the mirror sites reachable at http://www.qmail.org . Below
are some things I found before I encountered that list.
A Perl script which converts a user blah's /var/spool/mail/blah
mailspool mailbox to a Maildir in their user directory is Russel
Nelson's 1996 script:
http://www.qmail.org/convert-and-create
More on "convert-and-create" below.
Sidebar on Procmail's "formail" program:
According to discussion on the Postfix mailing list,
around 1 July 2001, there is also apparently a program which comes with
procmail, called "formail" which with the command "formail
-s" will convert from mbox mailboxes to maildir format. But formail
does not seem to do this. It's -s option splits a
mailbox file (containing any number of emails) on stdin into separate
emails, but all on stdout. However, I received an email on how to
use formail
to achieve the conversion:
FYI if you use procmail, or probably
even if you don't, this works:
First, change your procmail config to
Maildir mailboxes (add / to end of mailbox path).
Then, something like:
mv $MBOX /tmp/mbox ; mkdir $MBOX ;
formail -c procmail < /tmp/mbox && rm /tmp/mbox
will re-deliver all messages in $MBOX and
convert the mailbox to maildir.
There is also a perl script called "mbox2maildir" - which is
based on create-and-convert. Qmail has a perl script and a C
program complement each other. The links are to http://www.qmail.org .
I don't need maildir2mbox right now. It is 4.5 k of code,
with a total of four words in two comments. For a substantial
program with conditional statements nested 6 deep, this is exemplifies
the extreme reluctance of some programmers to write comments in their
code to assist mere mortals - including probably themselves a short
time later.
I was considering using mbox2maildir. Its doco is
self-contained
Usage: mbox2maildir uses the same environment
variables as maildir2mbox:
MAILDIR is the name of your maildir directory;
MAIL is the name of your mbox file;
MAILTMP is ignored.
MAIL is deleted after the conversion.
WARNING: there is no locking; don't run more than one of
these! you have been warned.
When downloading, don't use a Windows machine! It will change all
the 0A newlines into 0D 0A and this screws up everything.
I had to change its shell specification on the first line from #!/usr/local/bin/perl
to #!/usr/bin/perl
However, I did not use mbox2maildir, except a its key lines of
code, because I created >>> mb2md
<<< .
Ensuring new user accounts have a ~/Maildir/
Some changes need to be made to the system normally used for creating
new user accounts. In RH7.1, the /usr/sbin/useradd binary
(pointed to by /usr/sbin/adduser) uses a config file /etc/default/useradd
which points to the directory /etc/skel/ which is the
prototype for user directories.
To make this happen, as root, from /etc/skel/ I gave the
command:
maildirmake Maildir
to create a Maildir format directory and subdirectories called /etc/skel/Maildir/
.
Now running useradd blah -p xyz creates the appropriate
directories in /home/blah/, and but a file /var/spool/blah
was still created by the useradd binary. As far as I know, this
is not a problem. Postfix still delivers via Maildrop to the
above Maildir directory.
Backup and Reporting Mailbox Size etc.
(Added 2002 October 23.)
I run a nightly cron job to back all my user directories and their mail
to another Linux machine on the LAN. When I was using Mbox
mailboxes, which could be altered at any time with incoming mail, I
found I could not simply tar-gzip the directories theselves, but had to
copy them to a temporary directory and tar-gzip them there.
Probably, with Maildir mailboxes, I don't need to do this, but it is
part of the backup script and I still use it.
As time goes on and mail accumulates, mailboxes grow and this poses
some problems:
- Storage limits on the server's hard drive.
- Time taken to copy and tar-gzip for backup purposes.
- Total size of the entire backup file - I want to keep it under
700 Megs so it will go on a CD-R.
Therefore, mailboxes need to be emptied or deleted - but
which ones to get rid of? In order to decide, I wanted to know
how much storage each mailbox was taking in the tar-gzip file.
Here is a shell script - my slight modification of an excellent script
by Michael Carmak on the Courier
Users list on 23 September 2002. Thanks Michael!
maildir-report-3.sh.txt
maildir-report-3.sh.txt.gz
An example of its output follows. It does not report the final
size of the entire backup file, since it does not create any such file.
It generates temporary tarballs of each mailbox, and reports on
the size of that, amongst other things. In the final backup
file, the size would be different, but not by much. So with this,
I can see which mailboxes contribute most to the backup file size.
This script reports on one user only, and it reports on all the
mailboxes except for the Inbox. To do that, I would need to
modify
it to look into /cur as well. But I have a pretty good
idea
of what is in the Inbox since I am using it all the time. What I
want
to know about is the mailing list maiboxes, the virus and spam pits,
and
also any mailboxes I had forgotten about which are growing fat, such as
ones
to keep messages prior to filtering, the Drafts mailbox etc.
These are a selection of my 210 or so mailboxes.
Mailbox:
.0-Inbox-Old.Inbox-02-03
Number of
files:
1447
Total size of all
files: 21.0592 MB
Disk space
used:
25M
Approx tar.gz
size:
13M
Mailbox:
.0-Inbox-Old.Post-Filter-Inbox
Number of
files:
9888
Total size of all
files: 294.157 MB
Disk space
used:
326M
Approx tar.gz
size:
107M
Mailbox:
.0-SPAM-etc.0SPAM-HTML-refresh
Number of
files:
7
Total size of all
files: 56.7617 kB
Disk space
used:
88k
Approx tar.gz
size:
29k
Mailbox:
.Drafts
Number of
files:
1397
Total size of all
files: 22.4293 MB
Disk space
used:
26M
Approx tar.gz
size:
6.1M
Mailbox:
.Lists.Mail-Courier-users
Number of
files:
5697
Total size of all
files: 21.7214 MB
Disk space
used:
30M
Approx tar.gz
size:
3.8M
Mailbox:
.Lists.Music-dsp
Number of
files:
14324
Total size of all
files: 43.9091 MB
Disk space
used:
67M
Approx tar.gz
size:
10M
Mailbox:
.-Sent
Number of
files:
3096
Total size of all
files: 37.2873 MB
Disk space
used:
46M
Approx tar.gz
size:
22M
This is basically the end of this installation
story. Please see the parent directory for how I installed
SqWebMail and
Postman web-mail programs (with extensive mods to Postman). Also
there
you will find a long, long page on how I installed IMP in the middle of
2000. Now, a year later, I was going to install IMP again, but I
eventually opted for Postman instead.
Below is a link to an account, gradually degenerating, of my attempt to
install IMP 2.2.3 in June/July 2001.
I hope it helps someone - especially the IMP crew to improve their
installation documentation. That was the purpose of my previous
IMP page - and it is one of the most popular pages on my site. It
has helped people - such as on chap who says my page was an essential
step in him providing web-mail for a bunch of high-schools in Germany.
Trying to installing IMP - and giving up due to PHP problems
Note!
This documents a partial and unsuccessful attempt to install IMP.
I am documenting it here in the hope that it will help some
people install IMP and in particular help someone improve the IMP
documentation.
I do not want to attempt to improve the documentation myself, as Chuck
of
the Horde/IMP team has suggested I should, because I don't know enough
about
IMP and I think that documentation should be maintained in a cohesive
manner
by one or a few people with a long-term involvement in the project.
As time goes by, the value of what lies below - from June/July 2001 -
will hopefully diminish.
IMP is a highly respected Web-mail program, with a wide user base,
under active development, with new features being added and with
constant attention to security matters.
On 15 October 2001, I received the following email which makes me think
that RPM versions of IMP may be easier to install:
First of all thanx for the tutorial, it helped
me
starting the mail server on my computer. I also wanted IMP on my
computer
and I discovered that horde now has rpm packages for both horde and
imp.
Now there aren't any problems with configuring anymore... (I had the
same
problems as you had).
Simply install them and read what you need to change (a
few minor settings). If your still intrested in IMP then I can tell you
how I did
install it. The rpm packages are avaidable from:
ftp.horde.org dir: /pub/RPMS/noarch/rh7
The problem I had below was in following the Horde/IMP documentation
to install the PHP things I needed as a preliminary to
installing
Horde/IMP. I say things because I knew nothing about PHP,
all the various library files in various places (directly as part of
Apache
and elsewhere) and whatever other PHP related files for IMAP, and
whatever
Horde-modified version were requred to work with IMP. I had no
interest
in PHP and no knowledge - and in this condition, with my best efforts
to
follow the Horde/IMP documentation, my PHP things installation
work stopped Apache from booting.
Hopefully the problems I faced have been resolved with new RPMs and
documentation, but I haven't tried. If anyone does find a
straightforward path to installing IMP on a stock standard Red Hat 7.1
system, without having any prior knowledge of PHP, then please let me
know the precise URLs of the all the RPMs, tarballs and documentation
files you used. I will list them here and consign what lies below
to an historical footnote in a separate file.
|
I have moved the IMP stuff to a separate directory: IMP/
Update history
- 2001 August 10: Initial page put on site. (A partial page had
been at another URL.)
- 2001 August 11: Updated what I wrote about restarting
Postfix after changing the aliases database.
- 2001 September 30: Added notes about /etc/maildroprc and
RPMs for Mandrake Linux.
- 2001 October 4: Added note from Daniel Stone about XFS
being faster than Reiser for IMAP.
- 2001 October 9: Added note about keeping Mozilla happy with
MAXPERIP to be greater than 4.
- 2001 October 17: Added pointer to the Postfix Virtual
Domain Howto. Added a note about how the IMP installation may be
easier now, but I haven't tried. Moved the IMP material to a
sudirectory.
- 2001 November 13: Fixed errant ">" in "rpm -qf
<file name>" command.
- 2001 December 9: Updated the mb2md section to point to the list
of mbox to Maildire conversion programs at www.qmail.org.
- 2002 October 20: Modified the block diagram and added a
section under it on filtering for viruses and spam.
- 2002 October 23: Added script for estimating backup storage
requirements.
- 2003 June 1: Page put out to pasture since I have done a
new
one for recent versions of Red Hat, Postfix, Courier IMAP and Maildrop.
Added
a note about files for IMAP and POP3 in /etc/pam.d/ .