Setup

1. If you don’t already have one, create an account with AWS.
2. Take note of your access key. You will need to place it in the script in order to connect to the AWS EC2 API.
3. Create an Amazon EC2 key pair. You need this to launch and connect to your EC2 instance. Download the private key and store in on your system. In my example, I have the private key stored at /home/takaitra/.ec2/takaitra-aws-key.pem
4. Create an EBS volume in your preferred zone (location). Make sure it is large enough to store your backups.
5. Create a security group called “rsync” that allows connections on two inbound TCP ports: 22 (for SSH) and 873 (for rsync).
6. Ensure a recent version of Python and Boto are installed on your system. In Debian, this is accomplished by running the command ‘apt-get install python-boto’

The Script

The below script automates the entire backup process via boto (A Python interface to AWS). Make sure to configure the VOLUME_ID, ZONE and BACKUP_DIRS variables with your own values. Also update SSH_OPTS to point to the private key of your EC2 key pair. <aws access key> and <aws secret key> need to be filled in on line 19.

#!/usr/bin/env python

import os
from boto.ec2.connection import EC2Connection
import time

IMAGE           = 'ami-8e1fece7' # Basic 64-bit Amazon Linux AMI
KEY_NAME        = 'takaitra-key'
INSTANCE_TYPE   = 't1.micro'
VOLUME_ID       = 'vol-########'
ZONE            = 'us-east-1a' # Availability zone must match the volume's
SECURITY_GROUPS = ['rsync'] # Security group allows SSH
SSH_OPTS        = '-o StrictHostKeyChecking=no -i /home/takaitra/.ec2/takaitra-aws-key.pem'
BACKUP_DIRS     = ['/etc', '/opt/', '/root', '/home', '/usr/local', '/var/www']
DEVICE          = '/dev/sdh'

# Create the EC2 instance
print 'Starting an EC2 instance of type {0} with image {1}'.format(INSTANCE_TYPE, IMAGE)
conn = EC2Connection('<aws access key>', '<aws secret key>')
reservation = conn.run_instances(IMAGE, instance_type=INSTANCE_TYPE, key_name=KEY_NAME, placement=ZONE, security_groups=SECURITY_GROUPS)
instance = reservation.instances[0]
time.sleep(10) # Sleep so Amazon recognizes the new instance
while not instance.update() == 'running':
    time.sleep(3) # Let the instance start up
time.sleep(10) # Still feeling sleepy
print 'Started the instance: {0}'.format(instance.dns_name)

# Attach and mount the backup volume
print 'Attaching volume {0} to device {1}'.format(VOLUME_ID, DEVICE)
volume = conn.get_all_volumes(volume_ids=[VOLUME_ID])[0]
volumestatus = volume.attach(instance.id, DEVICE)
while not volume.status == 'in-use':
    time.sleep(3) # Wait for the volume to attach
    volume.update()
time.sleep(10) # Still feeling sleepy
print 'Volume is attached'
os.system("ssh -t {0} ec2-user@{1} \"sudo mkdir /mnt/data-store && sudo mount {2} /mnt/data-store\"".format(SSH_OPTS, instance.dns_name, DEVICE))

# Rsync
print 'Beginning rsync'
for backup_dir in BACKUP_DIRS:
    os.system("sudo rsync -e \"ssh {0}\" -avz --delete {2} ec2-user@{1}:/mnt/data-store{2}".format(SSH_OPTS, instance.dns_name, backup_dir))
print 'Rsync complete'

# Unmount and detach the volume, terminate the instance
print 'Unmounting and detaching volume'
os.system("ssh -t {0} ec2-user@{1} \"sudo umount /mnt/data-store\"".format(SSH_OPTS, instance.dns_name))
volume.detach()
while not volume.status == 'available':
    time.sleep(3) # Wait for the volume to detatch
    volume.update()
print 'Volume is detatched'
print 'Stopping instance'
instance.stop()

A current Debian system with an MTA installed and working. This guide is specific to Exim4 although many of the steps would apply to other MTA’s as well. Also, the guide assumes you store your mail in Maildir format. If you store your mail in single files (mbox), you will have to adjust some of the commands below.

Install Packages

Use aptitude or apt-get to install the packages ‘spamassassin’ and ‘sa-exim.’

Configuration

Create a file called .forward in your home directory with the following content. If needed, update the save command to point to the location of your junk/spam directory.

# Exim filter
if $h_X-Spam-Status: CONTAINS "Yes"
or
$h_X-Spam-Flag: CONTAINS "Yes"
then
save $home/Maildir/.Junk/
finish
endif

Edit /etc/exim4/sa-exim.conf and comment out the second SAEximRunCond attribute like so:

# Remove or comment out the following line to enable sa-exim
#SAEximRunCond: 0

Edit /etc/default/spamassassin and change the ENABLED flag to 1.

Finally, start spamassassin and configure it to start at boot if needed:

# /etc/init.d/spamassassin start
# update-rc.d spamassassin defaults

Optimization and Maintenence

Take a look at the log file /var/log/exim4/mainlog. You should see SpamAssassin doing its thing for any new emails coming in. You will see what spam ranking it assigns to the message and what its fate is (allowed, allowed but flagged, or permanently rejected).

A good way to increase the accuracy of SpamAssasin is to teach it. First, organize your spam and non-spam (which we will call “ham”) into separate folders. Try to make sure you don’t miscategorize any. For this to work well, you will need over a hundred of each type of email–the more the better. Then, run the ‘sa-learn’ command on the folders. For example, assuming your ham is in your inbox and your spam is in a folder called Junk:

$ sa-learn --ham --showdots /home/username/Maildir/cur/*
$ sa-learn --spam --showdots /home/username/Maildir/.Junk/cur/*

I recently installed Mailman on on my server to provide a mailing list for my extended family. While in the end, I was able to scrounge up the articles I needed by searching the web, many of them were woefully outdated. Here is a short article that pulls together my research and describes in one place what is needed to get Mailman running happily under Debian etch with Exim4.

Prerequisites

This guide assumes that you are running a recent release of Debian and have Exim4 installed and working.

Installing and Configuring Mailman

To install mailman, simply run the following command:

apt-get install mailman

During the install, you will be prompted to choose which languages you want mailman to support.

After the install is complete, follow the instructions given during the install and setup the Mailman-specific mailing list.

newlist mailman

There are just a few changes that must be made to the basic configuration. Open /etc/mailman/mm_cfg.py and edit the following items:

# Default domain for email addresses of newly created mailing lists
DEFAULT_EMAIL_HOST = 'list.example.org'

# Default host for the web interface of newly created mailing lists
DEFAULT_URL_HOST   = 'list.example.org'

# Uncomment this. In this setup, the alias file won't need to be changed.
MTA=None   # Misnomer, suppresses alias output on newlist

The last line makes no functional changes to mailman but will stop commands like “newlist” from outputing messages we won’t need. Restart mailman so that the configuration changes take effect:

/etc/init.d/mailman restart

Now would be a good time to set up any other mailing lists you will need using the same “newlist” command. If your list will be using anything other than the DEFAULT_URL_HOST we set up earlier as its web interface hostname, make sure to pass that to newlist with the -u flag.

Exim Configuration

The classic way of integrating Mailman with your MTA is to add each mailing list address to /etc/alias as a pipe to the mailman process. This is no longer the recommended way to configure Mailman with Exim. In fact, when I did try to add a piped alias, Exim choked on it because its default configuration no longer allows these for security reasons. So instead of adding dozens of lines to our alias file, we will be following the exim.org how-to to allow all Mailman addresses to automatically be handled by Exim.

Assuming you are using the split config, you will need to create the files listed below. If you are using a single file for configuration, you will need to find the appropriate places to insert the items.

/etc/exim4/conf.d/main/04_mailman_options:

# Mailman macro definitions

# Home dir for the Mailman installation
MM_HOME=/var/lib/mailman

# User and group for Mailman
MM_UID=list
MM_GID=list

#
# Domains that your lists are in - colon separated list
# you may wish to add these into local_domains as well
domainlist mm_domains=list.example.org

# The path of the Mailman mail wrapper script
MM_WRAP=MM_HOME/mail/mailman
#
# The path of the list config file (used as a required file when
# verifying list addresses)
MM_LISTCHK=MM_HOME/lists/${lc::$local_part}/config.pck

/etc/exim4/conf.d/router/450_mailman_aliases:

mailman_router:
driver = accept
domains = +mm_domains
require_files = MM_LISTCHK
local_part_suffix_optional
local_part_suffix = -admin : \
-bounces   : -bounces+*  : \
-confirm   : -confirm+*  : \
-join      : -leave      : \
-owner     : -request    : \
-subscribe : -unsubscribe
transport = mailman_transport

/etc/exim4/conf.d/transport/40_mailman_pipe:

mailman_transport:
driver = pipe
command = MM_WRAP \
'${if def:local_part_suffix \
{${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
{post}}' \
$local_part
current_directory = MM_HOME
home_directory = MM_HOME
user = MM_UID
group = MM_GID

After you finish creating the various configuration files, run the following commands to build the updated configuration file and restart exim:

update-exim4.conf
/etc/init.d/exim4 restart

Apache Configuration

mailman uses CGI to create a web interface for its mailing lists. We need to configure Apache in order to get this piece working. First create a file to store some new aliases for the web server.

/etc/apache2/conf.d/mailman:

Alias /pipermail /var/lib/mailman/archives/public
Alias /images/mailman /usr/share/images/mailman
<directory /var/lib/mailman/archives/public>
DirectoryIndex index.html
</directory>

Then create (or edit) a VirtualHost entry to allow the scripts to run.

/etc/apache2/sites-available/list.example.org:

<virtualhost *:80>
ServerName list.example.org
ServerAdmin webmaster@list.example.org
DocumentRoot /var/www/
<directory /var/www/>
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Order allow,deny
allow from all
# This directive allows us to have apache2's default start page
# in /apache2-default/, but still have / go to the right place
RedirectMatch ^/$ /cgi-bin/mailman/listinfo
</directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</directory>
</virtualhost>

If this is a new file, remember to symlink it to the sites-enabled directory.

Finally, restart Apache so that the changes take effect.

/etc/init.d/apache2 restart

Administer your List

That completes the setup! You can begin administering your new list at http://list.example.org/cgi-bin/mailman/listinfo