poptoimap

poptoimap fetches mail from a POP3 server and delivers it to an IMAP server after optional filtering. It is distributed under the terms of the GPL.

Download

poptoimap 0.2.0 is a single python script.

Rationale

For years I've downloaded my mail with fetchmail and delivered it through an MTA (usually exim) to an MDA - procmail or maildrop - which applies spam checking and filters the mail into folders based on criteria such as which mailing list it comes from. I wanted to combine this with the flexibility of reading my mail from an IMAP server and was surprised to find that many IMAP servers don't specify any way to deliver mail to them other than through the IMAP protocol, and neither of the common MDAs support IMAP destinations. It would be easy to write a generic IMAP delivery tool but I decided to go further and combine fetching, filtering and delivery into one program, saving the overheard of reloading regex scripts and opening a new IMAP connection for every message.

poptoimap is written in python, using its standard libraries to reduce development effort, and allowing its control file, poptoimaprc, to be implemented in python, offering a great deal of flexibility with a clean and simple syntax.

What about messages from your own system?

Such messages don't usually pass through a POP server, but you can still arrange for poptoimap to process them. All you have to do is have them delivered to your default system mailbox (you may need to remove your .procmailrc or .mailfilter files if you've been using procmail or maildrop), install a simple POP server, and get poptoimap to fetch from this mailbox as well as from your ISP. It's even easier than it sounds.

Command-line usage

You'll need to write a poptoimaprc file to be able to use poptoimap, but as that can be as simple or complicated as you like, I'll get the command line options out of the way first.

usage: poptoimap [options]

Options
--versionShow program's version number and exit.
 
-h,Show this help message and exit.
--help
 
-v,Print some extra information.
--verbose
 
-c RCFILE, Use an alternative config file instead of the default.
--rcfile=RCFILE
 
-r, Create IMAP folders if they don't exist.
--create

poptoimaprc

poptoimap's control file is usually called poptoimaprc. Its default location is $XDG_CONFIG_HOME/realh.co.uk/poptoimaprc. If $XDG_CONFIG_HOME is unset its default value of ~/.config is used, where ~ is your home directory, thus the full path is ~/.config/realh.co.uk/poptoimaprc. If this file doesn't exist poptoimap looks for ~/.poptoimaprc instead. If that in turn doesn't exist each branch of $XDG_CONFIG_DIRS (whose default value only has one branch, /etc/xdg) is checked for realh.co.uk/poptoimaprc ie by default it would look for /etc/xdg/realh.co.uk/poptoimaprc.

In other words in 99% of cases you should call the file ~/.config/realh.co.uk/poptoimaprc if you want to do it the neat way, or ~/.poptoimaprc for the old-fashioned way.

Alternative files may be specified with the -c or --rcfile options. The ~ character is acceptable at the start of a filename.

Whatever it's called the file is executed by poptoimap as a python script.

A very brief introduction to python

python has a clean and simple syntax, which is easy to learn, but it has one important difference from most other languages.

Indentation

Whereas it's usual to delimit logical blocks of a program (eg a function body, if-clause, loop) using some sort of brackets, python uses indentation. Each line in a logical block must begin with the same combination of spaces and/or tabs, and a child of that block must use the same indentation plus at least one extra space or tab. The exact combination of spaces or tabs doesn't matter, but it must be consistent within each block. I like to use a single tab per level with my text editor's tab stop set to 4 spaces instead of the common default of 8. The line introducing a new block, eg a function definition or if statement, (nearly?) always ends with a colon. Semi-colons are not used to separate statements, just newlines.

Strings

Strings can either be enclosed with single or double quotes. Additionally, multiline strings can be enclosed in triple quotes ie 3 double quote characters. Comments start with a # character and continue to the end of a line.

The letter r immediately before the opening quote of a string means the string is "raw", in which case backslashes are interpreted as literal backslashes instead of for escaping control characters. You will find this handy for writing regular expressions where strings often contain literal backslashes to escape regex special characters.

Example

def my_function(arg):
    """ Function body.
    A (pydoc) string at the start of a block
    describes the block. """
    if arg == True:
        print "'Double-quoted' string"
    else:
        print r'"Single-quoted" string'
    # Execution continues here in either case
    return another_function(arg)

.poptoimaprc essentials

The least you must do is specify the details for logging onto your IMAP server and at least one POP3 mailbox. The IMAP server is specified by setting the IMAP_SERVER variable to a URI string, and a POP mailbox by using poptoimap's pop() function eg:

IMAP_SERVER = 'imap://tony:secret@localhost'
pop('localhost', None, 'tony', 'secret')

By default emails are all delivered to the IMAP folder called 'INBOX' but you may override this by assigning a folder name string to the DEFAULT_FOLDER variable. Ways of changing the folder for each POP server or depending on the content of each message (like procmail) are described below.

IMAP URIs

An IMAP URI string begins with 'imap://' or 'imaps://', the latter meaning IMAP over SSL. The next part of the string is an optional user details section, separated from the hostname by an at ('@') character. The user details may contain just a user name or a user name followed by the password, separated by a colon (':'). If no password is given you will be prompted; if no user name is given, the current user's login name is used. The hostname or IP address may optionally be followed by the port number, separated by a colon. The default ports are 143 for imap and 993 for imaps.

pop()

You may specify more than one POP3 mailbox to fetch mail from in turn, by using poptoimap's pop() function repeatedly. It takes up to 6 arguments:

def pop(host, port, user, passwd, ssl, folder):

All these arguments are optional and python offers a number of ways of specifying only a few of them. If you wanted to specify only the user and password, two possible ways of doing this are:

    pop(None, None, 'tony', 'secret')

or

    pop(user = 'tony', passwd = 'secret')

The default host name is 'localhost', and the defaults for user and passwd are the same as for IMAP_SERVER. ssl takes a boolean value: whether or not to use SSL, defaulting to False. port defaults to 110 or 995 depending on ssl. You may also specify a (default) IMAP folder name to use for emails from this POP mailbox instead of 'INBOX' or DEFAULT_FOLDER by using the folder argument.

Filtering with deliver()

You may provide a filtering function by defining a function called deliver in your .poptoimparc. It takes one argument which is a python list (array) in which each element is one line of the current message as a string terminated by a newline character. Normally you will not need to use this argument but it is provided just in case. deliver() is called once for each message fetched whereas the rest of .poptoimaprc is only called once each time poptoimap is run.

Due to the way the python environment is set up anything your .poptoimaprc script defines outside of deliver() will not be available to deliver(). Helper functions may be defined inside deliver() though, and you can use "static" variables by redefining the delivery object as discussed below.

Functions available to deliver()

deliver()'s job is to run a series of tests on each message and decide where to deliver it (or not). poptoimap makes some functions available to deliver() to make this easy.

match(expression, case_sensitive = False, match = 'header')

Tests each line of the message against expression and returns a match object if any of them match it, or False if there's no match. Usually you only need to test whether the result is False or not.

expression is usually a python regular expression string; you may use a precompiled regex object instead, but this is only really practical in advanced cases where you redefine the delivery object.

case_sensitive is self-explanatory.

match may be 'header' (the default), in which case only header lines are tested, 'body', in which case only lines in the body of the message are tested, or 'all' to test all lines.

to(destination)

to() is the main function for the final stage in delivering each message. Calling to() sends the current message to the given destination. destination is usually an IMAP folder name. It may be omitted to use the default folder name, either for the current POP mailbox or DEFAULT_FOLDER or 'INBOX' as described above.

There are two special cases of destination other than IMAP folders. If the string begins with a '|' character, the rest of the string is treated as a command to which the message is piped on its standard input (stdin). If the string begins with '!' the rest of the string is an email address (or more than one, separated by whitespace) to which the message is forwarded.

Once to() is called, the message is considered finished with; an exception is raised to prevent the rest of your deliver() function being run. Normally exceptions are used to handle errors, but they may also be used to handle "benign" events like this. poptoimap traps the exception, deletes the current message from the POP server and passes on to the next message.

In case none of your conditions are met and deliver() completes without calling to() or kill(), the message will be delivered to the default folder.

cc(destination)

cc() is similar to to() but does not consider the message finished with and no exception is raised. Your deliver() function will continue to execute for the same message from that point.

kill()

Leaves the message undelivered, raising an exception which is treated similarly to the one raised by to(). This is a good way to deal with messages you are quite sure are spam.

xfilter(args, shell = True)

Pipes the message through an external program before continuing. The message is piped to the command's stdin and read back from its stdout. args may be a single string or a python list (array) representing the command and its arguments. If shell is False the command is run directly, otherwise it is run in the default shell. See python's subprocess module for more details of how the command is called.

Note that although the message may be altered by the filter, the argument passed to deliver() is a copy and will not be changed.

size()

Returns the size in bytes of the current message. For a line count, call len(message) where message is the argument of your deliver() function.

replace_message(message)

Replaces the current message with the one passed as the argument. This must be in the same format as the message passed in to your deliver function: a python list of lines including trailing newlines.

Redefining the delivery object

This is an advanced topic which may be added here later. Even I haven't had to go that far despite having some very complex rules.

Example poptoimaprc

This is my complete poptoimaprc with all its arcane rules. Sensitive information has been masked.