E-Mail Handling: Setting up an E-Mail server

WARNING

This guide has NOT been thoroughly tested. This was made from combining some various notes of an initial setup that seemed like it was working rather okay-ish. This guide is currently being shared for possible reference, in case it ends up being at all helpful, but this is not a guide that has been thoroughly reviewed. Consider this to be “in development”, and not yet recommended for anyone expecting/demanding a sufficiently tested guide.

This notice is expected to be reviewed after some further review/testing (which might not happen for weeks).

This notice was made approximately September 10, 2013.


Introduction/Preparation

About this guide

Here is an overview of what to expect from this guide.

Things to expect

This guide performs the following:

Set up a working E-Mail server

Be able to receive E-Mail from the Internet.

Be able to send E-Mail to the Internet.

Perform other sensible tasks, such as being able to provide remote access to the E-Mail. (May not be part of this guide yet.)

Automatically assign spamocity scores

Automatically assign scores to E-Mails. These scores indicate the likelihood of the E-Mails being spam.

  • “Spam” is basically being defined as E-Mail that is either unsolicited and containing:
    • commercial advertisements
    • malware
    • phishing-style attack
  • Keeping the trend of using pork-related ternminology, other types of E-Mail are referred to as “ham”.
Sorting E-Mails for people

Perform some sorting mechanisms, so that suspected spam can be seperated from other E-Mail

Implement desired priorities for receiving E-Mail

As described by some other sections there are some different potential priorities. E-Mail which is vaguely suspected of being spam can be saved, or completely blocked.

Why simple?

This tutorial was designed to show people some simple steps for techniques that evalutate E-Mails for anti-spam efforts. A lot of people may frequently equate the concept of “advanced” and “state of the art” as being the opposite of “simple”. So, why is this guide aiming for simple anti-spam measures?

Easier entry

Those who are experienced spam fighters may be more inclined to pursue options to be more aggressive. For people who have no experience with anti-spam yet, they may have more questions about how to deal with problems. This approach is designed to minimize problems so that people can, most comfortably, get their feet wet.

Reliable fallback plan

People can try more elaborate approaches, which may involve changing more configuration options or even adjusting which software, or hardware device, is proccessing the results of incoming SMTP traffic.

Keeping directions simple may have the benefit of keeping the directions fairly stable, so the technique can be effective in a wider variety of circumstances (including new changes that may made by attackers). Having an available simple setup also provides a nice option to be able to fall back to in case more elaborate attempts do not end up delivering the desired performance/payoff.

Some popele may desire to more time and effort into dreams of using more experimental techniques for detecting and handling spam. Even people willing to make such investments may have less stress when knowing how they can reverse the process, if necessary.

This guide does not...

And, here is a list of things to not anticipate.

The guide does not:

Use a chroot for security

A future version of the guide might. Using a chroot is recognized as a way to limit the severity of problems. However, they may provide additional limitations, and might also take some additional time to set up.

Maximum protection of resouces

The current version of this guide will, by default, not implement a heavy focus on preventing spammers from using up resources such as:

  • disk space
  • network bandwidth
  • and CPU usage

Instead, the focuses are on the following goals, in order:

  1. not making irreversible mistakes
  2. trying to reduce the time that people waste dealing with spam (including reading any of it)
  3. making life easy for legitimate senders to send E-Mail

(If making life easy for legitimate senders was more important than reducing time that people are wasting by dealing with spam, then simply not using any anti-spam procedures at all could achieve that goal.)

Some people may wish to be more aggressive, which may limit how many resources get taken up. So, the guide may point out options on how people can choose to explore these options, if desired. However, the default steps, which are specific steps that are shown to work, do not do this much.

The reason for not minimize resource usage is this: failure to be able to receive the message can result in an irrecoverable false negative. Hopefully false negatives may be rare, but the decision was made to try hard to make any such situation recoverable. Blocking the E-Mail early, by rejecting it in the original SMTP connection, may minimize resources spent, but also prevents the ability to dig up the E-Mail in case the automated routines end up miscategorizing the E-Mail.

Trying to attack spammers

Claims have been made about sputtering (sending feedback/characters slowly) being an effective way to cost spammers resources, slowing down their effective rate. Article on spammer attacking

This guide does not try to attack spammers. Wasting the time of spammers also impacts the flow of normal E-Mail, and may be fruitless (by not achieving the desired effect) against spammers who are capable of following standards about retrying the sending of E-Mails. Greylisting may effectively hurt certain attacks who don't follow the standards, including attackers who can only send E-Mails briefly, but these techniques would not stop attackers who have un-brief opportunities to send E-Mail. Attackers who have large botnets of “zombie” computers may often be able to get around these styles of defenses (simply by acting like a normal E-Mail server would act).

Network setup

For this setup, we used the following configuration. (Using this precise configuration is fairly unrelated to the topic of setting up the E-Mail server. These details probably won't matter much at all, but the detials are being provided in case they end up being helpful, or even in case they just help satisfy some curiosity.)

A physical machine ran a virtual machine. (The first real technical step on the physical machine was: Installing an operating system.) For a guide on creating a virtual machine, see: making a virtual machine.)

Main firewall services were provided by a separate virtual machine. This virtual machine was created using some older guides (including the older “general guide” from the tutorial on making a virtual machine.)

The physical machine really had just three jobs, two of which were related to this task. Those two jobs were running virtual machines, and redirecting traffic to the main firewall. (The third task was acting as a fileserver. Some people may start objecting to this design already, feeling that fileserver activities should be occurring on a different server. However, that is a topic that is quite unrelated to the main content of this guide, and so is not being discussed further here. People who don't like the described setup are free to create their own network design.)

After firewalling was set up, this E-Mail server was set up on a second virtual machine. (Creating that was done using a newer, more specific walkthrough which is at making a virtual machine.)

Machine prepared

The main things to note about the E-Mail server is that it was sufficiently prepared: the operating system was installed, some basic configuration (to harden the system) were performed, networking was operational (as proven by the ability to remotely access the system), and remote access was set up. The automatic addressing (DHCP/IPv4) server and name resolution (DNS) servers were configured with details about this server. (Since this was a virtual machine being added to an existing network, all of these details were pretty well covered by the specific walkthrough section of the tutorial on making a virtual machine.) In addition to these features, which should be considered some of the basic necessities, some additional configuration has been applied, such as running a file integrity checker. All of these tasks should be covered by the tutorial on making a virtual machine (and specifically the section about making a child image) and/or the tutorial on setting up an installed operating system.

The following decisions/customizations were made, and documented:

A name was assigned to the (virtual) machine
host name rules
A number was assigned to the (virtual) machine

Picked a two digit number that was not being used by any other virtual machines that might run on this same host.

Details on the purpose/reason for doing this are discussed in the section about: virtual machine number

System requirements were determined

An operating system was selected. For RAM, the decision was made considering the system requirements for the operating system (e.g. operating systems), the E-Mail software (which might be rather neglibile, although Microsoft Exchange may have some notably high requirements), and Anti-Virus software (e.g. ClamAV Memory Requirements). While most OpenBSD virtual machines may be happy with about 272MB of RAM if they are not running Anti-Virus software, for this machine a bit more memory is called for. (We'll try 640MB although 768MB or 1024MB or 1040MB might be more sensible, especially if clamd has grown over time since the time of this writing.)

Time

Warning: Setting up and configuring an E-Mail server might generally take a bit longer than the servers for some other network services. For instance, a DNS server can initially be set up to forward traffic, making it a bit useful while minimizing requirements for local configuration. A DHCP/IPv4 may require little more than identifying subnets in a single configuration file and making sure the server is started. In contrast, E-Mail servers need to know about what user names are accepted, where E-Mail data gets stored, so there is a bit more configuration that is needed. Also, at least some of the software implementations generally require more more steps to set up. Hopefully this last point will change as technology keeps improving, but, at least historically, this has definitely been true in some cases. (FWIW, Wikipedia's article on qmail: section called “simplicity” notes, “For common configurations, qmail is significantly easier to configure and deploy.”)

Deciding on an MTA

Some choices include OpenSMTPD, SendMail, Postfix, Zimbra, Microsoft Exchange, or perhaps some others (see: Alternatives article (page 1 of 2), 9 options for running an MTA).

The solution chosen (for the first version of this guide) is SendMail. One benefit of SendMail is that SendMail has been included with many different operating systems, pre-installed, including OpenBSD (at least up through version 5.3, which is the latest version of OpenBSD at the time of this writing in August of 2013, A.D.). Second, for better or for worse, the author has had some prior experience with SendMail, so this provided an opportunity to polish up some prior documentation.

Note that many people have disliked SendMail. One often-cited reason is because SendMail has historically had some major security problems. The security-conscious people behind OpenBSD (and OpenSSH) used to have a question about the “known insecure” reputation that SendMail generated for itself. The response stated, “Sendmail has had an imperfect security record, however the Sendmail authors and maintainers have been very receptive to reworking their code to make it much more secure (and this is a sadly uncommon response). The recent security history of Sendmail is not much different than some of the supposedly "more secure" alternatives.” This is quoted from OpenBSD FAQ 1.11: Why some specific software products are included and others are not (as archived by the Wayback Machine @ Archive.org from April 24, 2013). However, the quoted text did get removed later in the year (perhaps in anticipation of defending Sendmail less, and moving to OpenSMTPD?)

OpenBSD FAQ: Why some specific software products are included and others are not discusses some of the alternative software. Here are some additional brief blurbs about some alternatives:

Usage notes

E-Soft/“Security Space” Mail Survey from January 1, 2012 queried nearly 2 million servers (plus another third of a million which did not respond), and reported the following:

Top servers: Exim (43.32%), Postfix (23.61%), Sendmail (12.43%), Microsoft (12.12%)

Survey plots show that from the start of 2006, Microsoft looked like it hovered around 20% until about April of 2009, when usage started declining at about the same rate as SendMail. SendMail, Exim, and Postfix had continued trends rather steadily from 2006 through the end of 2011.

Postfix

Along the way, several guides were found that use Postfix. So, even if Postfix may not currently be covered in as much detail here, this seems to be a quite viable option with a number of pre-existing guides that may be followed. This guide may expand to cover Postfix in more detail at a later time.

Note that Postfix uses the IBM Public License. Wikipedia's page on IBM Public License describes it as free, but then discusses some issues that people have. OpenBSD FAQ: Why some specific software products are included and others are not states that the software “can not be considered” for inclusion in the product, because of the license.

Linux Professional Institute: summaries of changes between versions 3 and 4 of LPIC-2 and LPIC-3 certification: section on MTA coverage on LPIC-2 states, “Sendmail has been dropped to the awareness level in favour of Postfix. With the presence of Sendmail dropping to under 13% of publicly accessible mail servers, while Postfix accounts for approximately 23% of the publicly accessible mail servers, only Postfix configuration will be covered.” (The statistics seem to match the E-Soft survey quoted earlier in this guide.)

OpenSMTPD

Some heavy experimentation with OpenSMTPD was attempted, but OpenSMTPD ended up being largely passed on, for the first version of this guide. There were two reasons for this decision. One is that attempting to set up virtual users did not seem to offer the same flexibility as a known alternative. The other reason is that preliminary research indicated that other solutions had better integration with some of the anti-spam solutions. When creating this guide, here was a desire to explore support for many of the anti-spam solutions. Although OpenSMTPD might have been able to somehow work with anti-spam solutions, the method of using OpenSMTPD appeared to require some additional research. A desired deadline was impending, so the desire was to focus research time on starting by just trying to get the anti-spam tools working.

This may be reviewed later, but do understand that research is likely going to be required for support of virtual users, and reserach is likely going to be required for implementing content filtering/handling.

Status of filter support, from September, 2013

BSD Now, Episode 3: “MX with TT” (starting at 9 minutes) have people interviewing the creators of this software. About 14min7sec, various people in the interview noted, “we rely on the fact that if a tool can as SMTP proxy” ... “it should work”. “It should make it easy to make it compatible with a lot of things”... “very easy to setup a kind of loopback with any program that just allow re-inject of mail in the queue. So, that's pretty much everything can work that way, even if not probably the most efficient in some situation. At least you can interface quite easily.” In other words, the software really doesn't provide these sort of features, but it follows standards so that it can work well with other software that may provide these features. (The reason this isn't too useful is because other software that can communicate with SMTP might just be able to handle mail directly, so there may be less need to be using OpenSMTPD in that configuration.

They mention filtering with anti-spam solutions, again, about 18min/18min30sec into the video. 19min20sec notes the functionailty is “mostly in place” but the developers have been careful to make sure things aren't being broken.

QMail

Wikipedia's article on “qmail”: section called “Copyright status” notes about Daniel Julius Bernstein's product, “qmail is the only broadly deployed MTA in the public domain.” That does make qmail look like a highly attractive alternative, and it will likely be explored further at a later time. The biggest concern about qmail is an unsubstantiated belief that its behavior may be less standardized, even though there may be beneficial reasons (like being able to be a trendsetter). So, starting with a more widely deployed option seemed like a possibly more logical approach for an initial setup. Whether this “ubstantiated belief” is really true or not can be documented later, after it has been more fully explored.

Solutions provided by Microsoft
IIS's SMTP support

Windows XP Pro (not Windows XP Home Edition), Windows Server 2000, and Windows Server 2003 may come with support for an SMTP server. (A solution that is not by Microsoft, Windows SMTP Server's SourceForge page, is described as follows: “Since later versions of Windows (e.g. Windows 7) do not include the SMTP store-and-forward client, this application project creates a Windows service to offer similar services”

This guide is not providing many details on that solution, at this time. MS KB 837851 may provide some resources. Technet: Configure SMTP E-Mail (IIS 7) may be updated information related to newer operating systems.

Microsoft Exchange

A component called the “Categorizer” may help to sort mail. Third party products may interface with this particular component of Microsoft Exchange. If the mail logs indicate that the mail got to the categorizer, but then did not make it to the end user, that can often indicate the mail got handled by content filtering. (This might be somewhat old information, from Exchange 2003. That might, or might not, also be fairly true with newer versions.)

No further details about Microsoft Exchange are provided here at this time.

Others
Exim

This software just seems to not be discussed by setup guides as much as older alternatives like SendMail and Postfix. However, the software is the default MTA for Debian. Secure Space Mail Survey from January 1, 2012 shows Exim had 43.32% of usage share.

hMailServer version 4

Version 4 was Open Source. The company has closed the source code for version 5 of the software.

XMail

GPLv2. Not mentioned by Wikipedia's comparison of E-Mail servers. Voted highly by StackOverflow question on SMTP server (misplaced question that belonged on Server Fault or another site).

Action

Installing the MTA

Before trying to install an MTA, check if it may be pre-installed.

If it does need to be installed, some generalized guidance for installing software may be found from the section about software installation. (No further details on “installing” the software are provided here, meaning that there are no further details about how to get the software copied onto a computer so that the software may be run. However, there are details available here on configuring the software once it is installed.)

Running the MTA
Overview/Resources/References

The general topic is covered by the section on Servers getting E-Mails, which provides details about various software solutions.

Start the software

Get the software started. The following may provide information about specific implementations

Starting SendMail

More specifically, Basic sendmail program execution.

Configuration needed: receiving E-Mail
Listening to the needed network interface(s) and/or IP address(es)

Make sure that the program receiving mail will listen to traffic on the network interface cards that the software needs to be listening to. (Some E-Mail programs might default to only listening to the loopback interface. Such a default needs to be changed.)

Having SendMail receive mail
Overview

For SendMail, this usually involves specifying a pre-existing configuration file with that option, and it is worth noting that directly modifying the configuration files used by the program is not the preferred way to make changes. (This gets discussed in more detail in the section on available sendmail main program configuration files.)

If using a pre-existing configuration file is not an option, then creating a customized configuration file is the way to achieve the desired goal. (The configuration files are not the simplest items to create from scratch, so most technicians will likely want to start by having a text editor open up a provided configuration file, and making customizations as desired.)

Steps to take
  • Know this recommendation: avoid directly editing the configuration files used by SendMail. (This is discussed further in the “Overview” section just before these steps to take.)
  • Identify which configuration file to use: see available sendmail main program configuration files
  • (The remaining steps probably won't be needed if using the example command lines shown by this guide to basic sendmail program execution. However, double-checking should not take long.)

    Adjust the “-CconfigFilePath ” command line option.

Supporting the desired domain(s)/network(s)

Make the E-Mail server recognize which domain(s) it should accept traffic for. The term “domain” probably refers to DNS domains (rather than an alternative like an “Active Directory domain”), and definitely is meant to refer to the portion of the E-Mail address that shows up after the @.

This might or might not be necessary, especially if “virtual user” addresses end up getting implemented later. However, this might also be a simple and fast step to get some initial E-Mail working, which many people will probably want to see. So, plan to do this.

  • Identify the domain(s) that E-Mail will be getting received at. (e.g.: example.com)
  • Also identify the IP address of the SMTP server (which is the computer being worked on). This may or may not be necessary/useful. To find that information, see: getting current network address.
  • Apply the information as appropriate for the E-Mail server, using the following directions as much as they may be useful.
Supporting domains in SendMail

The names might need to be added to a file that SendMail uses, such as the /etc/mail/local-hostnames file. See: sendmail's Supported Domains.

Supporting the needed user account(s)/address(es) (@ the supported domain(s)/network(s))

Specify which users should be receiving mail.

This may be related to the role of an MDA which helps to get E-Mail where it needs to go (which will typically be in users' mailboxes). Specific details for specific program(s) may be available. As a generalization, information useful for accomplishing this task may be found in the section about MDA details.

SendMail

There is quite a bit of information available in the section on using SendMail as an MDA.

For easy/quick testing for a single E-Mail address or two, the “Local mailbox” method (mentioned by the section on using SendMail as an MDA) may be the fastest to initially implement.

The “virtuser” method offers more flexibility, with a notable cost simply being that it takes a bit more time to initially set up support for the /etc/virtual-domains file. Supporting “virtuser” can quite likely be the nicer end result, so that process is recommended for people willing to invest a bit of time to achieve such a preferred result. To do this, follow the steps for setting up SendMail's support for the /etc/virtual-domains file.

Simple scenarios

As noted by the SendMail virtual table configuration file section:

For organizations with extremely simple configurations, leaving the configuration file completely blank might actually be a perfectly fine option. (If the “virtusertable” file does not indicate how E-Mail should be handled, then the MTA software may just try some other mail-handling methods like using E-Mail aliases or local mailboxes.
Supporting more elaborate mail handling

If there actually is some sort of more elaborate setup desired, the detailed instructions (which may require some reading, comprehension, and thought) may be found by using SendMail as an MDA: section on supporting “virtuser”. That provides some details about various possibilities.

Note that changes to these files do not take immediate effect; more work is needed. Further details are provided after samples of these configuration files.

Sample rules

Here is a fairly quick sample. This shows some of the capabilities: it may be exactly what some people are looking for, and entirely unrelated to what other people are looking for.

The contents of the text file need to be customized. (The .test is just meant to make this example an invalid domain until things are properly customized. Replace all of “example.com.test” and the other demonstrating domain names with domain names that would actually be used. Similarly, customize usernames as appropriate.)

cpytobak /etc/mail/virtual-domains
cpytobak /etc/mail/virtusertable
echo example.com.test | sudo -n tee -a /etc/mail/virtual-domains
echo sample.example.com.test | sudo -n tee -a /etc/mail/virtual-domains
echo user@example.com.test address@example.net.test | sudo -n tee -a /etc/mail/virtusertable
echo @sample.example.com.test demo@example.org.test | sudo -n tee -a /etc/mail/virtusertable

(Note that the above commands are sending output to two different files.)

The above configurations show two rules being implemented. The first is meant to forward all E-Mails to receive all E-Mail addressed to user@example.com.test and send it to address@example.org.test. The second example rule receives all E-Mails to any address @sample.example.com.test and forwards all such E-Mails to demo@example.org.test.

For further details on what is possible, see:

Once the configuration file has been updated, it still won't take effect until a couple of tasks are performed:

Current status:

At this point, an SMTP client on the local host (such as telnet, as shown by SMTP sample) should be able to send E-Mail to users on the system. If E-Mail clients are permitted to run on the E-Mail server (which might be undesirable, perhaps mainly from a standpoint of wishing to reduce clutter on the E-Mail server), then using other E-Mail client software on the server may be an option. Use software installation process to install some decent/pleasant software, if needed. Some options may include:

  • For text-mode Unix: Something like Pine (re-alpine, or Alpine for older systems, or pine for even older systems), or mutt.
    • Users of Microsoft Windows may find a solution at PC-Alpine (system requirements for these releases were not documented.)
  • For graphical environments: IceDove, or IceApe, or Mozilla Thunderbird, or SeaMonkey
  • Other options: E-Mail MUA program, Wikipedia's comparison of E-Mail clients.
Troubleshooting

If an E-Mail client tried to send E-Mail, but the E-Mail did not seem to be received and /var/log/maillog shows “ stat=Deferred: Connection refused by [127.0.0.1]” (or some other IP address for the same system), then make sure that the TCP (tcp or tcp6) ports are being listened to: check out the results of the netstat command as shown by starting SendMail.

If the /var/log/maillog shows “ stat=Service unavailable”, look to see if there are any lines that say something like “NOQUEUE: tcpwrappers (localhost, 127.0.0.1) rejection”. If so, review: Allowing okay SendMail traffic.

Enabling connectivity

Internet connectivity part #1: adjust firewalls.

Internet connectivity part #2: Fortunately, this is probably not needed in many cases. If you are on a connection using a dynamic IPv4 address (historically much more common using “dial-up” connectivity), the MTA may need to be configured to send mail to the ISP's MSA. (At least in OpenBSD's SendMail, this is called “SMART_HOST”, as described by the /usr/share/sendmail/cf/openbsd-proto.mc file; see the section on how to update SendMail *.mc and *.cf configuration files.)

Forwarding/Replying

Set up E-Mail forwarding/relaying/etc. if desired. (Perhaps see: forwarding E-Mail.) (These may be some fairly old directions... careful review may be warranted...)

Content Handling
Initial overview

The basic steps might include:

  • Analyzing the contents of E-Mail to figure out if action should be taken
  • Deciding what to do with the E-Mail
  • Taking action

This guide provides details on a setup that involves using the following software:

ClamAV

This software is used to determine whether the E-Mail contains malware on it

SpamAssassin

This software is used to try to determine whether an E-Mail is “spam” (known to be unwanted) or “ham” (E-Mail that should be sent to a normal E-Mail box).

AMaViS's successor, Amavis (using amavisd-new)

This software provides two nice things:

  • Speed: AMaViSd-new probably results in scans being completed faster, due to techniques like keeping scanning software stored in memory
  • Ease of setup: Installing AMaViSd-new and enabling support for ClamAV and SpamAssassin may be easier than many other ways of trying to get ClamAV or SpamAssassin to work. Also, additional scanners can be added to AMaViSd-new fairly easily.

Wikipedia's article on Amavis notes that license is mostly GPLv2, with some BSD-style licensing thrown in. (The BSD-style licensing is less restrictive than GPLv2, which is less restrictive than GPLv3.)

The name of this program may be easier to remember by keeping in mind that the original version's spelling of AMaViS stood for “A Mail and Virus scanner”.

The setup with SendMail is a bit complicated, based on the included documentation from the /usr/local/share/doc/amavisd-new/README.sendmail-dual file.

Procmail

This software is used to sort E-Mail. A simple way this software may be used is to have E-Mail marked as being likely spam to automatically go into a folder that is meant to store such E-Mail.

Basic software installation

If AMaViS is going to be installed, install it first. It is believed that: The AMaViS package might depend on some other software, such as ClamAV, and so other software might end up not requiring separate installation.

Anti-Virus
Installation

If ClamAV was not installed as part of AMaViS, then install ClamAV separately. See: software installation.

Setting up updates

Definition files files need to be updated. Details on setting this up maybe found at: guide to making a virtual machine: section on initial setup for ClamAV for Unix.

Scanning incoming E-Mail

ClamAV may come with support for a “milter” (a SendMail filter), which is named clamav-milter. However, this guide is simply going to rely on AMaViS to run ClamAV scans.

SpamAssassin
Installation

If ClamAV was not installed as part of AMaViS, then install ClamAV separately. See: software installation.

Scanning incoming E-Mail

This guide is currently going to simply rely on AMaViS to run SpamAssassin.

Expected results
headers added by SpamAssassin: Forum post states: “The default headers that” SpamAssassin “normally adds are:” “X-Spam-Flag”, “X-Spam-Status”, “X-Spam-Level”, “X-Spam-Checker-Version”
smtp-vilter

This looked like a possible approach to integrate SendMail with other mail-handling software. However, upon trying to run the software, “__guard_local” errors occurred. This seemed like an issue with the package that could be worked around by just changing some sort of compile options, or which could be fixed by making the appropriate modifications to source code. This was noticed with OpenBSD 5.3: hopefully future versions of the package fix this issue.

AMaViS
Overview

The approach taken by this guide is the one used by the /usr/local/share/doc/amavisd-new/README.sendmail-dual file. This will involve creating separate SendMail configuration files for the role of getting (a.k.a. “receive”, “RX”) E-Mail, and the role of sending (a.k.a. “transmit”, “TX”) E-Mail.

Installation

See: software installation

The following notes were taken when using OpenBSD's installation (after having the respository already set up).

sudo -i pkg_add -ivv amavisd-new
Software dependencies
RPM-related software

One of the dependencies is an RPM handler. There may be options: e.g. in OpenBSD 5.3:

Ambiguous: choose dependency for amavisd-new-2.8.0p0:
 a  0: rpm2cpio-1.3p2:
 1: rpm-3.0.6p6:
Your choice:

Recommended; choose the option related to “rpm2cpio” (which in this case is item number zero). Reason: Reviewing the information on these ports, OpenPorts.se information on rpm2cpio and OpenPorts.se information on rpm, it seems that rpm2cpio is the smaller/simpler approach. This does not seem like software where more extensive functionality will likely be particularly useful.

SPF package
Ambiguous: choose dependency for p5-Mail-SpamAssassin-3.3.2p6:
 a  0: p5-Mail-SPF-2.8.0:
 1: p5-Mail-SPF-Query-1.999.1p4:
Your choice:

No strong recommendation is being made here. Choose one. Both options seem fairly small: SPF is about 100K rather than SPF-query being about 50K. SPF (Sender Policy Framework) may be a fairly good thing, so speculation is that the larger package may someday be nice to use.

OpenPorts.se package details on p5-Mail-SPF-Query, OpenPorts.se package details on p5-Mail-SPF.

GNU Privacy Guard version 1
Ambiguous: choose dependency for p5-Mail-SpamAssassin-3.3.2p6:
 a  0: gnupg-1.4.13:
 1: gnupg-1.4.13-card-ldap:
Your choice:

Recommendation: Choose the second option, since LDAP may be useful for understanding some types of user accounts.

OpenPorts.se package details on GNU Privacy Guard version 1

Some available documentation

The package may come void of any man pages, but other documentation is available. There may be a number of available pieces of documentation. The OpenBSD package installs several documentation files to /usr/local/share/doc/avavisd-new/ including:

AAAREADME.first
...
RELEASE_NOTES
...
README.sendmail-dual
...
README.sendmail-dual.old
...
README.milter

The file itself identifies itself as old; refers to an amavis-milter command (which has a -v option for verbose) However, this package does not come with amavis-milter

An early step is to determine the name of the socket file. For instance, in OpenBSD, although /usr/local/share/doc/amavisd-new/README.milter makes a reference to a /var/amavis/amavis-milter.sock file, the actual package may default to making a /var/amavisd/amavisd.sock file.

Other README.* files
...

Users of other packages may wish to check for these documents at that (/usr/local/share/doc/avavisd-new/) location. If the files are not there, such users may wish to use package management software to determine what files have been installed.

Information is also available online: amavisd-new Documentation files online contains the updated copies of the documentation files that get installed to the hard drive. Other resources may include: Amavis FAQ, amavisd-new Tips and FAQ, amavisd-new documentation bits and pieces.

Other files that may be found on the hard drive: /usr/local/share/examples/amavisd-new/amavisd.conf
/usr/local/share/examples/amavisd-new/amavisd.conf-default
/usr/local/share/snmp/mibs/AMAVIS-MIB.txt

Checking user, location (e.g. creation, permissions)
User

Check if there is a user already made for the SendMail Message Submission Program.

grep msp: /etc/passwd

Note: that is simply one example command that might turn up desired results on some systems. There really is not just one universal command that is particularly likely to work on various types of systems that may come with different default user names. Manual review may be prudent. The user database may be viewed with:

echo ${EDITOR}
sudo vipw

e.g., in /etc/passwd may be:

smmsp:*:25:25:Sendmail Message Submission Program:/nonexistent:/sbin/nologin

Otherwise, perhaps see: adding a user

Location: Directory for a receive queue
sudo mkdir /var/spool/mqueue-rx
Permissions/ownership
sudo chown smmsp:smmsp /var/spool/mqueue-rx
sudo chmod 770 /var/spool/mqueue-rx

The following is not from any official documentation or other pre-written guide that was found, but since directory has group of wheel, it makes sense. Note that this is a different directory than what was just lookd at.

sudo ${SHELL} -c "eval ls -ld /var/spool/mqueue*"
sudo chmod g+rwx /var/spool/mqueue
sudo ${SHELL} -c "eval ls -ld /var/spool/mqueue*"
Getting the SMTP server to work with AMaViS

Details are likely to vary between SMTP servers. Identify the correct section:

Getting SendMail to work with AMaViS
Creating desired SendMail configuration files for use with AMaViS

At the time of this writing, only one option is being documented here:

Using the “dual” setup

This section is heavily based on information that came from the /usr/local/share/doc/amavisd-new/README.sendmail-dual file.

Starting with templates

These are based on information that came from the /usr/local/share/doc/amavisd-new/README.sendmail-dual file.

Obtain and place the files using one of these methods:

Downloading and placing files

This method refers to an online resource that is only updated manually, if ever. So this method may be more prone to be out of date. However, this method is also simpler to perform than the alternative.

lynx -dump http://cyberpillar.com/dirsver/1/mainsite/techns/netfeats/messages/elecmail/svgeteml/emailrcv/sndmlrcv/amavismc/amvsmcnw.tgz | tee -a ~/amvsmcnw.tgz
mkdir ~/amavismc
tar -xzvvf ~/amvsmcnw.tgz -C ~/amavismc
echo ${CFDIR}
[ "X${CFDIR}" = "X" ] && export CFDIR=/usr/share/sendmail
echo ${CFDIR}
sudo cp -i ~/amavismc/usr/share/sendmail/cf/thishost-?x.mc ${CFDIR}/cf/.

(Once the files have been created, skip on down to the section about editing the configuration files.)

Extracting files from locally available documentation

The other method: manually extract the desired code from the /usr/local/share/doc/amavisd-new/README.sendmail-dual documentation files. Copy the file and then manually edit to remove the unnecessary lines.

cp -i /usr/local/share/doc/amavisd-new/README.sendmail-dual ~/thishost-rx-cutted-text.mc
cp -i ~/thishost-rx-cutted-text.mc ~/thishost-tx-cutted-text.mc
echo ${VISUAL}
${VISUAL} ~/thishost-rx-cutted-text.mc

... then, after making the needed manual edits:

${VISUAL} ~/thishost-tx-cutted-text.mc

... then, after making the needed manual edits:

sudo cp -i ~/thishost-rx-cutted-text.mc ${CFDIR}/cf/thishost-rx.mc
sudo cp -i ~/thishost-tx-cutted-text.mc ${CFDIR}/cf/thishost-tx.mc
ls -l ~/thishost-?x.mc

(Once the files have been created, skip on down to the section about editing the configuration files.)

Copy and paste

(Note: This is commentary. This is not a working method. Choose one of the other methods.)

The contents of the files are not showing up on a web page for easy copying and pasting. There are some tabs that could very easily be overlooked by copy-and-paste job.

Editing the configuration files
echo ${VISUAL} ${CFDIR}
[ "X${CFDIR}" = "X" ] && export CFDIR=/usr/share/sendmail
echo ${VISUAL} ${CFDIR}
sudo ${VISUAL} ${CFDIR}/cf/thishost-rx.mc

One item to know: the token “dnl” (which means “delete through newline”) is often used to eliminate text up through the following newline character. The lines that start with “dnl ” are comments.

One of the early lines, perhaps the third line of the file, states, “Insert here the usual .mc preamble, including OSTYPE and DOMAIN calls.” After that line, insert:

divert(-1)
#
# config file as described by cyberpillar.com
# based largely on the thishost-rx.mc file described by AMaViSd-new's
# /usr/local/share/doc/amavisd-new/README.sendmail-dual.txt file

# Note that lines beginning with "dnl" below are comments.

divert(0)dnl
VERSIONID(`@(#)thishost-rx.mc $Revision: 0.01 $')dnl
OSTYPE(openbsd)dnl
dnl
dnl If you have a non-static IP address you may wish to forward outgoing mail
dnl through your ISP's mail server to prevent matching one of the dialup
dnl DNS black holes. Just uncomment the following line and replace
dnl mail.myisp.net with the hostname of your ISP's mail server.
dnl
dnl define(`SMART_HOST', `mail.myisp.net')dnl
  • Note: Some of that content came from what was seen by ${CFDIR}/cf/openbsd-proto.mc file. Users of operating systems other than OpenBSD should probably use a different value for the “OSTYPE” line. (To see what might be better to use, try running: “ grep -i OSTYPE ${CFDIR}/cf/*.mc
  • Also, make the same sort of change to the thishost-tx.mc file, except replace the two occurrences of “thishost-rx.mc” with “thishost-tx.mc”. (The first of those two lines is in a “dnl” comment line, so does not impact the actions that the machine may take.)
Additional customizing options

Speculation: Customize the TCP ports used if desired: a default TCP port is TCP port 10024 for traffic (sent from sendmail MTA-RX) to amavisd-new listening on loopback interface. And, the other default TCP port is TCP port 10025 for traffic (sent from amavisd-new) to sendmail's MTA-TX which is listening on loopback interface.

[#amvismcf] Making *.cf files

Now we need to make the *.cf file(s).

Variable prep
[ "X${CFDIR}" = "X" ] && export CFDIR=/usr/share/sendmail
Identify the command to run (and more variable prep)

This documentation uses a custom variable that identifies the command that gets run. (This allows the documentation to apply to various systems with minimal required customization.)

In OpenBSD, the command to run is “ ${CFDIR}/m4/cf.m4 ”, so the variable may be set to:

export M4CFSCPT="m4 ${CFDIR}/m4/cf.m4"

The following does the same thing, but only if some checks prove to be valid:

[ -e "${CFDIR}/m4/cf.m4" ] && [ -x "$(which m4)" ] && [ -e "${CFDIR}/m4/cf.m4" ] && export M4CFSCPT="m4 ${CFDIR}/m4/cf.m4"

However, /usr/local/share/doc/amavisd-new/README.sendmail-dual indicates the command would be someting more like “ m4 ${CFDIR}/cf/m4/cf.m4

The following should only update the variable if the script is found at this alternate location:

[ ! -e "${CFDIR}/m4/cf.m4" ] && [ -x "$(which m4)" ] && [ -e "${CFDIR}/cf/m4/cf.m4" ] && export M4CFSCPT="m4 ${CFDIR}/cf/m4/cf.m4"
Creating /etc/mail/thishost-?x.cf files

The following example creates configuration files for the AMaViS/SendMail “dual” setup, and so creates two configuration files. Other approaches may need to adjust the input and output filename(s), and might only be creating one configuration file.

Note that the following tee commands do not specify -a, so these overwrites (and do not append).

m4 ${M4CFSCPT} ${CFDIR}/cf/thishost-rx.cf | sudo -n tee /etc/mail/sendmail-rx.cf
echo $?
m4 M4CFSCPT ${CFDIR}/cf/thishost-tx.cf | sudo -n tee /etc/mail/sendmail-tx.cf
echo $?

Optional: This is closer to what is recommended by AMaViS's README.sendmail-dual guide; it overwrites the default configuration. (Decide whether or not this is desirable.) (This is NOT needed when closely following only this guide. It might improve things (or break things) if trying to use a hodgepodge of commands from different guides. If following only this guide, don't bother (but don't worry if this was accidentally done).)

m4 ${M4CFSCPT} ${CFDIR}/cf/thishost-tx.cf | sudo -n tee /etc/mail/sendmail.cf
echo $?
Current status

At this point, the configuration files are created. For using the “dual” setup, there will be a need to shut down any existing copies of SendMail that are using old configuration files, and starting up a new copy of SendMail for each of the two configuration files that have just been created for this “dual”-style setup. Also, AMaViS should be started.

(Instructions on how to do all that are in a forthcoming section.)

If using other SMTP software

If none of the preceding sections, designed to be specific to certain SMTP server software, are applicable, then additional research may be needed. Check out AMaViS's documentation (using the documentation files from /usr/local/share/doc/amavisd-new/README.* or the amavisd-new Documentation files online which are largely the same files, although they may be different versions). Some additional tips are available at amavisd-new: “interfaces to MTA”. If AMaViS's documentation does not have suitable information in the bundled documentation, then the most official resource of information will be to check the documentation that came with the SMTP server software. (The topic of “milter” or “mail filter” may be helpful.) That approach may often be requiring research that is a bit more substantial than some other approaches. Perhaps easier, but more likely to rely on less official (and quite possibly incorrect, or theoretically even malicious) documentation is to just start searching the web for a guide.

Some setup details focused on AMaViS's amavisd
Configuring AMaViS's amavisd.conf file

The file that AMaViS generally uses will be the /etc/amavisd.conf file. There may be some additional sample files available. In OpenBSD, they are located in /usr/local/share/examples/amavisd-new/ but other operating systems may have the locations vary somewhat. The files in that alternate location are amavisd.conf, which describes itself as “a minimalistic configuration file” ... “with all necessary settings”, and a amavisd.conf-default which provides examples of “ALL CONFIGURATION VARIABLES WITH THEIR DEFAULT VALUES”. If there are troubles finding the sample files, try using: “ sudo find / -iname "amavisd.conf*" ”.

cpytobak /etc/amavisd.conf
echo ${VISUAL}
${VISUAL} /etc/amavisd.conf

There are quite a few recommendations for changes to this file.

Some items to change
Setting max servers

If using PostFix as the SMTP server, then $max_servers should match Postfix's amavisfeed configuration (as mentioned by CentOS HowTo guide for Amavisd).

(There are currently no corresponding directions for users of other SMTP servers.)

Set domain

Must change: $mydomain

Make it match the localhost's domain. (e.g., if the computer's name is compname.domname then the domain name is just the “domname” part)

Requirement for SpamAssassin

For SpamAssassin to work, you'll need to uncomment the following line. (This is done by removing the comment character from the start of the line. It is also customary (and might be required) to remove any white space from the very start of the de-commented line.)

# $helpers_home = "$MYHOME/var";  # working directory for SpamAssassin, -S
PID
Optional and recommended: uncomment the line referring to $pid_file
Verbosity

There is a line that, by default, states:

$log_level = 0;              # versbosity 0..5, -d

Set to 5 if desired. (For those who want to have lots of troubleshooting information, this is generally a good idea. For those who are looking for more speed, this may be something to try leaving set at zero, but keep this in mind in case troubleshooting is desired later.)

Simple recommendation: set to 5.

Networks

Find:

@mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
                  10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 );

(The first line shows loopback addresses and IPv6 addresses. The second line shows address ranges mentioned by IETF BCP5, which is the most well-known RFC number, RFC 1918.)

Once that text is found, add any IP address ranges that this network will be using. For example:

@mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
                  10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
                  [FD00::]/8
                  [2001:DB8::]/32 192.0.2.0/24                   198.51.100.0/24 203.0.113.0/24 );

Note: the FD00::/8 is a good address range to include. None of the rest of those newly-added examples should be used without being changed. They are only meant for examples (as described by RFC 3849 and RFC 5737). If any E-Mail actually specified those blocks, that may be improper/malicious, so do not actually use those examples. However, this shows how to do this if there are public address ranges to identify as your own networks.

Note: This did not just involve adding networks to new lines. The ); was also appropriately moved.)

tagging default

Since marking each message with a score may be a good idea, particularly on low-traffic servers and during the beginning when things are being tested, change:

$sa_tag_level_deflt  = 2.0;

... to:

$sa_tag_level_deflt  = -999;

(per CentOS HowTo guide for Amavisd... Scalix Wiki: SendMail with Amavisd: New HOWTO used -9999 )

Delivery Status Notification cutoff

Also change, from:

$sa_dsn_cutoff_level = 10;

... to:

$sa_dsn_cutoff_level = -998.9;

This identifies what spam level a DSN is not sent out for. E-Mail blacklist organizations consider DSN to be a nuisance, potentially useful to spammers, and therefore E-Mail that should be treated like spam. Therefore, sending out a DSN to the public Internet is rather unsafe. This setting prevents that from occurring just because of a belief that an E-Mail is spam.

Some commentary

SpamAssassin.Apache.org guide: “Scoring options” section refers to a deprecated but accepted option called required_hits for auto-deleting mail, which the site recommends not doing.

Official (or even clear) documentation has not been found about what the exact possible range of scores are, but SpamAssassin.Apache.org guide: “Scoring options” section does refer to “an integer or a real number.” (A “real” number just implying that it may have a decimal point.) Based on SpamAssassin.Apache.org guide: tests, there is a test that can add 1000.000 to a score.

It appears that these only affect site-wide rules. Users may override with ~/.spamassassin/user_prefs http://linuxgazette.net/105/youngman.html

More to change
Size limit

Also change, from:

$sa_mail_body_size_limit = 400*1024; # don't waste time on SA if mail is larger

... to:

$sa_mail_body_size_limit = 400*1024*1024; # don't waste time on SA if mail is larger

This assumes E-Mails don't have attachments over 400MB, which may be valid. Specifying a 400MB size in SA's configuration will be a size that is plenty large for the many E-Mail servers that limit E-Mails to a much smaller size, such as 10MB. The reason to increase is so that more files DO get checked. (That may not be desired if the server is overloaded.)

Many business people have been requesting a size limit larger than 10MB, as newer business documents (including word processing documents, and especially data files created by some popular “presentation software”) may often exceed 10MB. However, exceeding 400MB for each and every spam message may be a more significant bandwidth drain on spammers, and may be more prone to lead to tripping alarms. Therefore, spammers are probably not as likely to be exceeding that threshhold at this time (written in the year 2013).

More commentary: visibility

Note: At least some of the above values may be added to the E-Mail message headers. If spam info headers are being added, users can see the numeric values that get set to $sa_tag_level_deflt (which will be called “tagged_above”) and $sa_tag2_level_deflt (which will be called “required”. (Users will also be able to see what tests got applied.) This is not necessarily tragic, but server administrators often try to have information be done behind the scenes and so it can be comforting to be informed when information becomes potentially easily visible to an end user. (However, most end users will have header information be less prominent, and even most commonly invisible. So these details probably won't be seen by end users unless the MUA software highlights these details. However, that is just pure speculation: this example is not meant to imply that any popular software is known to highlight these details.)

More to change

The following is behavior selected because of the goal, which is simply to mark messages, and not to prevent delivery. This eliminates any possibility of mail becoming permanently unretrievable just because of a false positive, and allows system administrators to review spam (which could help with some tasks such as researching the contents of spam, possibly to help with better future spam detection).

People with different goals may not want to follow the next bit of advice. (Even better than just following this advice, or ignoring this advice, is to consider this advice and intelligently make an informed decision regarding what values would be preferred.)

Change from:

# $final_virus_destiny      = D_DISCARD;
# $final_banned_destiny     = D_DISCARD;
# $final_spam_destiny       = D_PASS;  #!!!  D_DISCARD / D_REJECT
# $final_bad_header_destiny = D_PASS;
# $bad_header_quarantine_method = undef;

... to:

## $final_virus_destiny      = D_DISCARD;
$final_virus_destiny      = D_PASS;
## $final_banned_destiny     = D_DISCARD;
$final_banned_destiny     = D_PASS;
# $final_spam_destiny       = D_PASS;  #!!!  D_DISCARD / D_REJECT
$final_spam_destiny       = D_PASS;  #!!!  D_DISCARD / D_REJECT
# $final_bad_header_destiny = D_PASS;
$final_bad_header_destiny = D_PASS;
## $bad_header_quarantine_method = undef;

(The reason for using a double-“comment character” is simply to help identify pre-existing comments.)

Additional required actions

Make any required directories that are referenced by now-uncommented lines in the configuration file.

sudo mkdir /var/amavisd/var
sudo mkdir /var/amavisd/tmp
sudo mkdir /var/amavisd/db
Handling support for ClamAV

To enable ClamAV:

  • Make sure ClamAV is installed. (Run: “ which clamscan ” or view a list of installed software. The software installation may describe how to do that.)
  • In amavisd.conf, change:

    # ### http://www.clamav.net/
    # ['ClamAV-clamd',
    #   \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.sock"],
    #   qr/\bOK$/m, qr/\bFOUND$/m,
    #   qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
    # # NOTE: .....

    and uncomment the lines that are meant to be uncommented when desired, so the result looks like:

    # ### http://www.clamav.net/
    ['ClamAV-clamd',
      \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.sock"],
      qr/\bOK$/m, qr/\bFOUND$/m,
      qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
    # # NOTE: run clamd under the same user as amavisd - or run it under its own
  • Also, make sure that the socket matches what is in ClamAV. To see what ClamAV is configured to use, get to a command prompt and run:

    grep -i LocalSocket /etc/clamd.conf

    Then, make sure that this filename is used as a socket by AMaViS:

    cpytobak /etc/amavisd.conf
    echo ${VISUAL}
    ${VISUAL} /etc/amavisd.conf

    (Locate "/var/run/clamav/clamd.sock" from this example, and replace that text with the actual filename specified by ClamAV's configuration.)

Ensure the specified E-Mail addresses are valid

Make sure the E-Mail addresses are valid, such as $virus_admin and $visus_admin_maps and $mailfrom_notifiy_spamadmin and others mentioned around there.

This probably means enabling support for virusalert@ and spam.police@

(This doesn't necessarily need to be done quite this early, but delaying this until too late may simply result in missing some E-Mails.)

A way to do that may be to add to /etc/mail/virtusertable and then update that database. e.g., make the file look something like:

virusalert@mycomputer.mydomain security@mycomputer.mydomain
spam.police@mycomputer.mydomain abuse@mycomputer.mydomain

(The names on the left side are some default E-Mail addresses used by AMaViS, while the names on the right side are commonly default aliases supported supported by the SMTP server.)

However, that involves adding support for virtual users. See: SendMail: virtual user.

If you take this route, after updating file, update the virtusertable.db database.

Also, if you've already started SendMail with the new config files, either have SendMail reload (SIGHUP ought to work) or quit those copies of SendMail (and restart).

[#smamavgo]: Starting stuff
Starting clamd

(An assumption here is that ClamAV has been pre-configured.

sudo freshclam -d
echo $?

(There might not have been a point to that... the error/return code has been known to be zero even when the ClamAV program is outdated and the software is unable to find clamd running.)

See: Starting clamd

[#smamavkl]: Stopping pre-existing SMTP Server and AMaViS
Stopping SendMail and Amavisd

(This doesn't stop clamd.)

First, try stopping the sendmail-rx. (Currently untested: Perhaps that will occur with /etc/rc.d/sendmail?)

However, there may be multiple SendMail instances, so the recommendation to safely stop all SendMail instances is to be fairly thorough.

The following works fairly well on OpenBSD. It may not work quite as well on some other platforms due to ps implementation differences and the lack of a pkill command.

sudo /etc/rc.d/sendmail stop
sudo time /etc/rc.d/amavisd stop
sudo ls -l /var/run/sendmail.pid
sudo head -1 /var/run/sendmail.pid
ps -auxww | grep -i sendmail grep -v grep
sudo pkill sendmail
ps -auxww | grep -i sendmail grep -v grep
Starting MTA-TX

(This might be particularly meant for describing the “dual” setup using SendMail and AMaViS. For configurations where one piece of software performs both roles of receiving E-Mail and also sending E-Mail, this is probably best to skip. That is, starting AMaViS is probably best to do before starting the SMTP software.)

The reason this gets started first is because we are trying to avoid a situation where another E-Mail component receives E-Mail, but then tries to pass the E-Mail onto this component, only to find out that there are problems communicating with this E-Mail component. The goal is to avoid causing problems for other internal E-Mail server components.

Start the MTA which will have the role of sending out E-Mail, which is being described as “MTA-TX”.

Using SendMail
Using the SendMail “dual” setup

If this guide has been followed the entire time, this may be done with:

sudo $(which /usr/sbin/sendmail ) -C/etc/mail/sendmail-tx.cf -L sm-mta-tx -bd -qp
echo $?
sudo cat /var/log/maillog

Unlike most programs that may accept a file path right after a parameter is specified, there is no space after the “ -C” and before the start of the path to the configuration file.

If the sendmail-tx.cf file file is not found...

If the process followed has been a hodgepodge between following this guide and following the README.sendmail-dual file, then the sendmail-tx.cf file might not exist. In that case, the following may work better:

sudo $(which /usr/sbin/sendmail ) -C/etc/mail/sendmail.cf -L sm-mta-tx -bd -qp
echo $?
sudo cat /var/log/maillog

Note that this sendmail.cf filename may be the name of a file that exists by default. The file probably does exist, even if it was not configured well for the “dual” setup.

Unlike most programs that may accept a file path right after a parameter is specified, there is no space after the “ -C” and before the start of the path to the configuration file.

Note: After starting the sm-mta-tx, don't start the sm-mta-rx yet.

Contrary to the order of instructions from amavisd-new package's /usr/local/share/doc/amavisd-new/README.sendmail-dual file, which goes starting RX and then TX and then amavisd, it makes sense to run these in reverse order. That way, software isn't trying to pass E-Mail to other software that isn't running yet. So, start the software in the order that is shown here, which is to start with TX and then AmavisD and then RX.

At this point, this server may be tested with telnet to port 10025 (using SMTP, as described by guide to manual communications using SMTP.

Check the queue to see what it is like before starting amavisd. This way, if there are any old/stuck messages, there will be less likelihood of thinking that AMaViS just delivered the E-Mail.

e.g.:

sudo mailq -C/etc/mail/sendmail-tx.cf

Unlike most programs that may accept a file path right after a parameter is specified, there is no space after the “ -C” and before the start of the path to the configuration file.

Starting AMaViS
Starting the server

By now, some basic configuration of /etc/amavisd.conf file should have already taken place.

Running the program normally

If you want to just start things the way they should normally be running, then run:

time sudo /etc/rc.d/amavisd start
echo $?
sudo cat /var/log/maillog
Debug mode
sudo amavisd debug

Note: with the “ debug” parameter, it does not exit. This lets you see amavisd's output. (Unfortunately, this has also been known to lead to a segmentation fault, even on a system where the program worked normally with high log verbosity when the program was started by being backgrounded.)

Queue check

Then, if MTA-TX has been started, see if amavisd generated any E-Mails complaining about the configuration:

sudo mailq -C/etc/mail/sendmail-tx.cf

Unlike most programs that may accept a file path right after a parameter is specified, there is no space after the “ -C” and before the start of the path to the configuration file.

(Note: At one time, checking that queue seemed like a good idea. Further reserach indicates that amavisd might not send E-Mails about problems... although that might appear to be true, Amavisd might have just been passing on older E-Mails that were stuck until amavisd started running. So checking the queue might not be quite as useful as previously thought when those instructions were made...)

Ownership check on initial run

This section is about supporting server executables being run as different users, and only applies if running the AMaViS server and also running the ClamAV server. (This is a common configuration, so this might apply to many people. These instructions might be needed, and probably will not cause harm if applied even when unneeded.)

The first time AMaViS is started, there is some configuration that is worth checking. (This checking is easier to perform after AMaViS is started.) Once this matter is correctly configured, there shouldn't be any need to re-check this every time that AMaViS is started.

First, check to see what usernames are running the AMaViS program and the ClamAV program. These example commands may need to be customized based on which ps version is being used, due to ps implementation differences.

ps -auxww | grep -i amavisd | grep -v grep
ps -auxww | grep -i clamd | grep -v grep

The first column shows the username that “owns” the running process.

Are those different users? If so, make sure that is sufficiently configured, or correct the situation as described by some comments found in a /etc/amavisd.conf file:

Have user running ClamAV user be in group of user that runs AMaViS

Make sure that the user running clamd is in a group that amavisd's user is in, such as a group named “_vscan”.

If that is not the case:

Place user in group
Ensure group exists

Ensure there is a group for virus scanning, or one even more specific for amavisd-new. First, check to see if there is a pre-existing group that fits that description.

grep -i vscan /etc/group
grep -i amavis /etc/group
grep -i scan /etc/group | grep -v vscan

If there is no such group, make one. (See creating a group.)

User account placement

Make sure that the user account running clamd is in such a group.

  • The process is quite likely to vary among different operating systems. In some cases, a good first step is to identify what group(s) the user is in first. (To do this, see sections about: identifying who is in a group, viewing what groups a user account is in.) The following is an example for OpenBSD:

    user info _clamav | grep ^groups
  • Perform the step that actually places the user in the group. (See: placing a user account into a group.) The process is quite likely to vary among different operating systems.

    Example(s)
    OpenBSD

    The following shows some commands that *might* work out okay on OpenBSD.

    Warning

    This example could also yank a user account out of a group that the user had been in, and there will probably not be a very convenient way of noticing that this happened. Later, there will probably not be a very convenient way of figuring out every group that the user has left. So, this could be quite undesirable. (The recommended action, for anyone who is not very familiar with these commands, is to take the time to read through the documentation showing the entire process of adding a group to the list of groups that a user is a part of.)

    Example command
    sudo usermod -G _clamav,_vscan _clamav
Restart ClamAV

Speculation: this might not be needed in some cases, if the user account has not changed (but if group membership did change). If a different user account is being used, then this is definitely needed.

That that the ClamAV server software needs to close entirely. (Note that even if the goal was to change the configuration file, this software documents its effects of SIGHUP and reloading the configuration file does not appear to be one of those effects. So a SIGHUP would not work to re-load the configuration file. However, the purpose of this restart is to change ownerships, not to simly reload its configuration files. So, fully closing down the software, and then restarting it up, is needed.)

Allowing ClamAV PID

Since prior instructions might have involved changes regarding the user account that runs ClamAV, re-visiting ClamAV PID file considerations may be worthwhile.

Allowing supplementary groups

ClamAV should allow supplementary groups. This is not done by default, but may have been done when following a guide to setting up ClamAV. Check out the configuration file: see ClamAV AllowSupplementaryGroups for details.

Stopping ClamAV server

Identify the PID used by ClamAV. (This may be done by running “ head -1 pidFile ”, using whatever customized value is appropriate for “pidFile”. If no PID file is being used, this can generally be figured out by using a technique described by the section of: viewing running software. For instance, the user account may be the very first column in ps output.)

Send a SIGTERM to that process. e.g.:

[ -f /var/run/clamd/clamd.pid ] && sudo kill $(sudo -n head -1 /var/run/clamd/clamd.pid)

The program might take a bit of time to close. Check its progress:

sudo tail -f /var/log/clamd.log

If the software has been running long enough, you'll probably see nine “SelfCheck: Database status OK.” spread out over 45 minute segments. (This is not in response to recently stopping the software, but is just from some old logged events.)

Then you'll see a “Waiting for all threads to finish” Look for: “--- Stopped at ” (followed by a timestamp).

When that is seen, press Ctrl-C to close “ tail -f fileName ”.

Starting ClamAV server

See: Starting clamd

Available communications

At this point, this server can be communicated with, by using telnet, by default using TCP port 10024.

Starting MTA-RX server
If using SendMail
If using the SendMail “dual” setup
sudo $(which /usr/sbin/sendmail ) -C/etc/mail/sendmail-rx.cf -L sm-mta-rx -bd -qp

Unlike most programs that may accept a file path right after a parameter is specified, there is no space after the “ -C” and before the start of the path to the configuration file.

echo $?
sudo cat /var/log/maillog
Also, start up a queue:
sudo $(which /usr/sbin/sendmail ) -Ac -L sm-msp-queue -q10m
echo $?
sudo cat /var/log/maillog
Additional commands to run
If using SendMail “dual” setup

(This may need to vary due to ps implementation differences.)

ps -auxww | grep -i sendmail | grep -v grep

alias mailq-rx='sudo mailq -C/etc/mail/sendmail-rx.cf'
alias mailq-tx='sudo mailq -C/etc/mail/sendmail-tx.cf'
alias sendmail-rx='sudo /usr/sbin/sendmail -C/etc/mail/sendmail-rx.cf'
alias sendmail-tx='sudo /usr/sbin/sendmail -C/etc/mail/sendmail-tx.cf'

mailq-rx

Unlike most programs that may accept a file path right after a parameter is specified, there is no space after the “ -C” and before the start of the path to the configuration file.

There should be no messages. If there are, and if you want those messages to be processed immediately, try:

sudo sendmail -C/etc/mail/sendmail-rx.cf -v -q
Ready for testing
Basic SMTP communication

At this point, this server may be tested with telnet to TCP ports 587 and 25. Even if firewalling is preventing communication from outside networks, testing may be done from the system (using the loopback interface). For directions about what communications to send over a successful connection, see SMTP sample.

Scoring

Send an E-Mail. The recipient should get the E-Mail and be able to see information from AMaViS in the hdears. As an example, the user may see something like this in the headers:

X-Virus-Scanned: amavisd-new at localnet
X-Spam-Flag: NO
X-Spam-Score: -0.5
X-Spam-Level:
X-Spam-Status: No, score=-0.5 tagged_above=-999 required=6.2
tests=[ALL_TRUSTED=-1, FROM_DOMAIN_NOVOWEL=0.5] autolearn=ham

Here is an explanation of the information shown by those headers: Apparently in the above example, the message came from a trusted source, so that adjusted the spam score to be -1. However, the domain did not have enough vowels to look likely legitimate (the name of the system used to generate this example was “opnsmtpd”), so that test did move the E-Mail +0.5 towards the direction of looking rather spam-like. The end result was a spam score of -0.5, which gave it a rating of X-Spam-Flag: NO. Similarly, the Spam-Status shows the score of -0.5 is below the required value of 6.2 for it to be considered spam. Therefore, the autolearn results was that this message is “ham”.

Internet communications
Adjusting network traffic

If communications are working internally, consider adjusting firewall rules to allow traffic from the Internet.

MX Record

If DNS is operating yet, adjust the DNS server configuration(s) to support an MX record.

Autostart

If you like what you see, then adjust the operating system startup processes so that these things happen automatically. (e.g., if this is a virtual machine, have the virtual machine start. Have the anti-virus software (clamd), scanning software (amavisd), and various SendMail pieces start (in the right order).

(Directions for this are not currently provided here...) See: automatically started files. The section on system startup configuration files may have information to develop an even classier approach.

Troubleshooting
Log (errorneously?) complains about anti-virus missing
Symptom

/var/log/maillog shows:

Date sysname avavis[PID]: (PID-01) (!)WARN: all primary virus scanners failed, considering backups

However, ClamAV is installed.

Solutions
Uncomment configuration

It seems that Amavisd tries to comment out support for all anti-virus software in the primary Amavisd configuration. Then, if no anti-virus support is found in the primary Amavisd configuration, it provides this warning and tries to rely on scanners that it tries as backup when the preferred method fails. The goal might be to get system administrators to uncomment just the configuration related to the software that will actually be used. Amavisd might be failing to use the preferred clamd but might still actually be scanning using a secondary command that it can use as backup.

ClamAV configuration may be commented out by default.

While uncommenting these lines, checking on the socket is probably worthwhile to do at the same time. (See the details provided by one of the other solutions in this section.)

Socket not matching/found

The filename of the socket that Amavisd looks for might not match the filename that ClamAV is using.

grep -i LocalSocket /etc/clamd.conf

Makes ure that is matching what is specified in the ClamAV section of the /etc/amavisd.conf file.

Perhaps clamd is not running.

Run it. Details may be found somewhere in the section about starting stuff.

Misc notes
An idea: Speeding up trusted communications:

Spam checking can slow down the process. On a (virtual) machine with insufficient RAM, preliminary log scanning indicated that an internal message took over 160,000ms (2 minutes 40 seconds). That was for delivering just one message.

Granted, that was on a machine that needed adjustment so that it wasn't swapping out the clamd program, and so that did need to be fixed, but the incident still led to thinking about allowing trusted users to skip the anti-spam.

This would probably be done cleanest by having sendmail-rx determine where the E-Mail is from (including whether the connection came in on the right IP address) and, if so, sending to the port of sendmail-tx.

These are simply some observations/speculations. No actual specific directions are being provided yet.

Manual page

man spamassassin

same as “ man 1 spamassassin ” ???

Tagging section of man page notes, “Note: before header modification and addition, all headers beginning with "X-Spam-" are removed to prevent spammer mischief and also to avoid potential problems caused by prior invocations of SpamAssassin.”

Adding tests
http://spamassassin.apache.org/tests_3_0_x.html perhaps see: http://www.yrex.com/spam/spamconfig.php

There are network tests available. http://wiki.apache.org/spamassassin/RazorAmavisd refers to a concept called “collaborative filtering network”. Note: At this time, the author of this text has not investigated thoroughly to see what kind of information ends up getting uploaded to the Internet. In other words, the privacy impacts of using these technologies have not been identified by the author of this text. If this is a concern, perform suitable research before deploying these tests.

See: Using SpamAsssassin Network Tests. Namely, it says to make sure that SpamAssassin is started without using -L and without using --local Then, check out the following sections: Also, make sure the individaul tests are enabled:

Bayesian
http://wiki.apisnetworks.com/index.php/SpamAssassin_Tutorial , http://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html#learning_options
use_bayes 1
bayes_auto_learn 1
Teaching the Bayesian filtering can be done with the sa-learn command: see http://linuxgazette.net/105/youngman.html but perhaps that is unnecessary when using bayes_auto_learn?
Using Vipul's Razor

http://wiki.apache.org/spamassassin/InstallingRazor states: “Note that Razor is not available for unlimited free use, so it is commented out in init.pre. It is currently free for personal use, subject to capacity constraints. See the Cloudmark SpamNet Service Policy for more details.” However: wiki.apache.org/spamassassin/UsingRazor#Questions_about_Razor_Licensing states, “The usage policy changed as of 3/13/2006 which makes the service freely available”.

http://razor.sf.net http://wiki.apache.org/spamassassin/RazorAmavisd

Make sure that Vipul's Razor is installed. Note that this might be pre-installed when SpamAssassin got installed, so checking for that may be faster than just trying to install. The name of the software package might be razor-agents

in /etc/mail/spamassassin/local.cf:

use_razor2 1 # http://wiki.apache.org/spamassassin/RazorAmavisd
razor_timeout 10 # per http://wiki.apache.org/spamassassin/UsingNetworkTests
	# use razor_timeout.  The value of 10 was selected to be similar to
	# an example seen at http://wiki.apache.org/spamassassin/UsingPyzor
# if using alt config file location then specify, e.g.
# razor_config = /etc/mail/spamassassin/.razor/razor-agents.conf
# as stated by http://wiki.apache.org/spamassassin/RazorSiteWide
# but not needed if using root's default home in /etc/razor/razor-agents.conf
# unless another location (e.g. non-root user home dir/.razor/) is overriding

Razor likely requires configuration to run properly:

Configure RAZOR:

Determine the account that Amavisd runs as. e.g.: _vscan or amavis

sudo -u _vscan razor-admin -v
sudo -u _vscan razor-admin -h

http://www.raygibson.net/kb/amavis/ suggests using “su” to that account, but that does not work for computer systems that properly prevent that account from being able to log in, as advised by ijs.si/software/amavisd#sec-host Furthermore, if that account has a home directory (e.g. if ~_vscan) that is /var/empty then we can't really count on storing files in a location related to that user. We'll store in /etc/razor as that is a location mentioned by the razor-agents manual page http://razor.sf.net/docs/doc.php?type=pod&name=razor-agents#configuration (when discussion the location of razor-agent.conf) Also, razor.sf.net/docs/doc.php?type=pod&name=razor-agent.conf#attributes says “The default is /etc/razor/ for root, and ~/.razor/ for every other user.”

sudo mkdir /etc/razor
sudo chown _vscan:_vscan /etc/razor
sudo chmod ug+rwx,o-rwx /etc/razor

Make some more configuration files

sudo -u _vscan razor-admin -d -create -home=/etc/razor | sudo -n -u _vscan tee -a /etc/razor/razormk.log

That should create a default /etc/razor/razor-agent.conf file. (That file may get located automatically, without needing to specify -home)

cpytobak /etc/razor/razor-agent.conf
echo razorhome = /etc/razor/ | sudo -n -u _vscan tee -a /etc/razor/razor-agent.conf

sudo -u _vscan cat /etc/razor/razor-agent.conf | grep -i razordiscovery

e.g.: razordiscovery         = discovery.razor.cloudmark.com If that does not show, try running:

sudo -u _vscan razor-admin -d -discover

razor-agent.conf

Verify operations: http://wiki.apache.org/spamassassin/RazorHowToTell

echo test >> /tmp/mailtest
cat /tmp/mailtest | spamassassin -t -D razor2 | head -1
expended output:
date stamp [PID] dbg: razor2: razor2 is available, version #.##
(I'm presuming that was a PID...)

Helping the razor database:
Details have not been fully documented here, but here are some pointers:
sudo -u _vscan razor-admin -register
man razor-admin

Pyzor

This package may need to be installed
sudo -i pkg_add -ivv pyzor

pyzor.sf.net

http://wiki.apache.org/spamassassin/UsingPyzor

in /etc/mail/spamassassin/local.cf: use_pyzor 1 # http://wiki.apache.org/spamassassin/UsingPyzor pyzor_options --homedir /etc/mail/spamassassin/pyzor pyzor_timeout 10 # http://wiki.apache.org/spamassassin/UsingPyzor # also mentioned by http://wiki.apache.org/spamassassin/UsingNetworkTests

Although http://wiki.apache.org/spamassassin/UsingPyzor indicated to use --homedir /etc/mail/spamassassin that seemed dumb, particularly since pyzor's filename of “servers” isn't pyzor-specific, and so would conflict with any other software related to spamassassin and which used such a generic filename. So, a /pyzor was added, which is why the abov eexample says /etc/mail/spamassassin/pyzor

If using /etc/mail/spamassassin/pyzor, then:
sudo mkdir /etc/mail/spamassassin/pyzor

Configuring pyzor: Let pyzor discover some servers that it can use later. sudo pyzor --homedir /etc/mail/spamassassin/pyzor discover output:


downloading servers from http://pyzor.sourceforge.net/cgi-bin/inform-servers-0-3-x

(e.g., the file there might say:) public.pyzor.org:24441

That may then show up in /etc/mail/spamassassin/pyzor/servers

cat /etc/mail/spamassassin/pyzor/servers

# per http://wiki.apache.org/spamassassin/RazorHowToTell / http://wiki.apache.org/spamassassin/UsingPyzor

echo test >> /tmp/mailtest

cat /tmp/mailtest | spamassassin -D pyzor 2>&1 | less

See: http://wiki.apache.org/spamassassin/UsingPyzor#Is_it_working.3F for details on what to expect.

DCC

Distributed Checksum Clearinghouse is another option. SpamAssassin's manual page for the SpamAssassin Mail Plugin for DCC states, “Note that DCC is disabled by default” ... “because it is not open source.”

rhyolite.com/dcc#availability / rhyolite.com/dcc#downloads does provide source code. However, although the source is public, it is not fully “open”. The license for free use of DCC seems designed to prohibit competition to the business of the copyright holder. So, some organizations are excluded. Also to absolutely require that people “provide corresponding data to other users.” (What that means is that even people operating private DCC servers must provide data to the public servers, so that others may benefit. This is a noble action, but the fact that this is required makes the software less flexible than typical “open source” software.

Because of these limitions, SpamAssassin defaults to having DCC support disabled. Also, this guide may provide only few details, and this may be less tested than some other options.

When public DCC servers may be used

rhyolite.com/dcc#client-problems indicates that selling the services of the publicly accessible DCC servers “has always been wrong” because of a couple of reasons: it is making money from another organization's bandwidth, and also profiting from the “human system administration work of the public DCC servers”.

http://wiki.apache.org/spamassassin/UsingNetworkTests says: “If you're a very large site, processing upwards of tens of thousands of messages a day, the DCC maintainers have requested that you consider setting up your own DCC server as described in dccd(8), and arrange to peer with the rest of the public servers, to reduce their load.”

rhyolite.com/dcc#client-problems mentions a higher threshold: “more than 100,000 messages per day”.

rhyolite.com/dcc#license states, “The non-commercial DCC software is distributed under a license that is free only to organizations that do not sell filtering devices or services except to their own users and that participate in the global DCC network. ISPs that use DCC to filter mail for their own users are intended to be covered by the free license.”

Setup

There may be few details here, but here are some pointers:

The firewall will need to be adjusted, as noted by wiki.apache.org/spamassassin/UsingDcc#I.27ve_installed_DCC_but_I_never_seem_to_get_DCC_hits.2C_even_though_I_should._Why.3F and rhyolite.com/dcc#client-problems

The support in SpamAssassin may also need to be modified. wiki.apache.org/spamassassin/UsingDcc#Enabling_DCC /etc/mail/spamassassin/v310.pre or perhaps the file is /etc/mail/spamassassin/init.pre as mentioned by http://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Plugin_DCC.html

Guide: guide

Other network tests
At the time of this writing, other tests are not mentioned by Using SpamAsssassin Network Tests. Post about SpamAssassin tests notes that some tests, like mime_defang, have been obsolete for years.
Whitelisting

There are options: edit local.cf with: spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html#whitelist_and_blacklist_options

Note that http://spamassassin.apache.org/tests_3_0_x.html shows that being in a user's whitelist, and being listed in an "all_spam_to" can both knock 100 off of the spam score. However, a test called GTUBE can add 1000.000 to a spam score. Although, that is an old list of tests: see: http://spamassassin.apache.org/tests.html

Further adjustments of results

Before writing your own SpamAssassin rules, remember that spammers are constantly trying to evolve their techniques. Creating an algorithm and then testing it and fine-tuning it to perfection is an effort that may be quite likely to be substantially beaten over time. So unless ongoing investment over time is expected, the effort made may have no long term payoff in effectiveness. Of course, rules could still be made for the fun of creating, but simply know that in the vast majority of cases, most people will probably not experience the desired payoff by implementing a theoretical, untested, brand new rule. Instead, relying on the evolving anti-spam efforts will be a far easier way to keep up with any evolving spam efforts. Learning how to adjust the settings of these rules may be a much better approach, so that the rules may evolve.

If custom rules are helpful, but then become less helpful over time, then a sensible approach may be to try the default settings (using newer versions of SpamAssassin rules). Perhaps that would be catastrophic: use your own judgement. At least in theory, it should be possible to duplicate mail and then handle each copy in two different ways, and see which mail-handling method provides the desired results.

Perhaps see: Fine-tuning SpamAssassn (article by Neil Youngman).

Other/older/misc notes
see also:
spamassassin --help
spamassassin --lint
echo $?

If that is non-zero then try:

echo ${PAGER}
spamassassin --lint -D | $PAGER

for more info.


No need for amavisd to run spamd: http://mail-archives.apache.org/mod_mbox/spamassassin-users/201001.mbox/%3C4B43FA52.40401@verizon.net%3E since that indicates that Amavisd creates a re-usable instance, perhaps restarting amavisd is good to do after updating SpamAssassin config? ijs.si/software/amavisd#features-performance