Exim

At the moment, this guide may have little detail about installing Exim. However, it does have some other details about how to interact with an installed server.

Despite the online version of this file having a filename and location including “rcv” (because this software can be used for receiving incoming E-Mail messages), this documentation currently covers Exim as software that handles sending outgoing messages, as well as receiving incoming messages.

[#eximiact]: Interacting with Exim

Unofficial man page for exim(8), made for Debian GNU/Linux, located at linux.die.net (on a site named after the singular version of the more famous plural word, dice) documents some various commands and parameters. However, there are some items to note:

  • The exim command seems to have multiple parameters that start with a single hyphen, and then contains multiple letters. So, this does NOT follow the common standard (perhaps commonly used with “getopts”) of using two hyphens before multi-letter parameters, and treating multiple letters after a single parameter as multiple parameters that are conveniently munged together. Such behavior is common to many Unix programs, but NOT the exim command. (The exim command's single-hyphen parameters are similar in behavior to the double-hyphen parameters recognized by many other commands, including the GNU release of many standard commands.)
  • Some of the commands mentioned on Unofficial man page for exim(8), made for Debian GNU/Linux, located at linux.die.net (on a site named after the singular version of the more famous plural word, dice) may not work. For instance, that man page mentions “mailq”, but that command has been found to not work on a machine using a popular software package named “cPanel” (which uses Exim).
  • One of the commands that should exist, and which is quite useful, is the “exiqgrep” command. Unfortunately, Unofficial man page for exim(8), made for Debian GNU/Linux, located at linux.die.net (on a site named after the singular version of the more famous plural word, dice) seems to lack some details that might be commonly supported (maybe just on newer versions of exiqgrep?)
    • Running “ exiqgrep -h ” should show some brief help.
    • In particular, it is heplful to understand that the -f parameter is expected to be followed by a space, and then a RegExp (which could just be plain text) which contains an E-Mail address surrounded by a less-than sign (“<”) at the start, and a greater-than sign (“>”) afterward. e.g.: “  -f <user@example.org>”
      • This is inconsistent with some other fields. Of particular note, the -r parameter also expects a RegExp that can show an E-Mail address, but the -r parameter should not be followed by a RegExp surrounded by inequality signs. Using the inequality signs (e.g., trying “-r <user@example.org>”) may simply result in an error message, as such improper usage effectively breaks the command. Instead, simply use “ -r user@example.org ”, without the “<” nor the “>”.)
      • When running “exiqgrep -h”, some of the output may show “sender address” “(field is "< >" wrapped)”. That looks much better than Unofficial man page for exim(8), made for Debian GNU/Linux, located at linux.die.net (on a site named after the singular version of the more famous plural word, dice). At the time of this writing, that website showed, “sender address (field is lq< >rq wrapped)”. Ugh...
  • Unofficial man page for the Exim Queue “grep” (“global ‘regular expression’ print”) command exiqgrep(8), made for Debian GNU/Linux, located at linux.die.net (on a site named after the singular version of the more famous plural word, dice)
[#eximtrak]: Tracking E-Mail

To find out what happened with an E-Mail message...

Gather message details

Determine as many of the following message details as possible:

  • What E-Mail address did the E-Mail message get sent from?
  • What recipient E-Mail address was the E-Mail message supposed to be delivered to?
  • When was the message delivery attempted?
  • What is the subject of the E-Mail message?

Actually, even a bit more may be helpful. See: E-Mail Tracking Helpful Data/Details.

Looking up details

You can use standard Unix tools like grep to look through log files. Exim comes with exigrep which, as understood by the author of this guide, appears to output all lines in the log file that are related to each message ID that shows up in matching text.

To re-iterate that in a bit of a less-condensed fashion,

  • Finds matching detail in a log
  • Determines the Message ID related to that line in the log
  • Outputs all lines related to each message ID that mat

Note: there is a somewhat similar-looking command name that does something different. So, exigrep is different from exiqgrep.

Examples

As an example, to search for E-Mail to a certain address:

exigrep "\<= .*sender[\@\\.]example\\.net\>? H=.* for whoisto[\@\\.]example\\.org.*>?$" /var/log/exim_mainlog
  • Customize the from address (“sender[\@\\.]example\\.net”. Note that the typical @ has been replaced by “[\@\\.]”. That is intentional, and should be done, in case a log line shows the E-Mail address slightly modified with a period.
  • Customize the recipient's address (“whoisto[\@\\.]example\\.org”. Note that the typical @ has been replaced by “[\@\\.]”. That is intentional, and should be done, in case a log line shows the E-Mail address slightly modified with a period.
  • The backslashes beofre the periods are optional. All they do is clarify an actual period, but if an un-escaped period is there then the program will simply match any character. Since only a period may be likely, that level of imprecision is probably not going to typically matter.

As an example, to search for E-Mail from a certain address:

exigrep "\<= .*sender[\@\.]example.net\>? H=.* for .+$" /var/log/exim_mainlog

As an example, to search for E-Mail to a certain address:

exigrep "\<= .* for whoisto[\@\.]example.org.*>?$" /var/log/exim_mainlog

If all you have is a subject line, you can search for T= (listing the message's topic).

If you didn't want to use exigrep then that's fine. Other commands like grep can be used to identify a line of text related to a message. Since chances are you'll want information from at least two lines related to each message, you might not find all of the details you want with a single grep command. However, what you do find will probably provide you with an Exim Message ID (which has 6 characters, then a hyphen, then 6 more characters, then a hyphen, and then two more characters). (Note that sometimes when you see an Exim Message ID, it may be part of a longer label. e.g., instead of Exim1M-essage-ID it might look like Exim1M-essage-ID-H where the -H indicates it is information related to a header.) Once you have a message ID, you can search for just that, and you get lines helpful for the message.

If a message is not being found, you may wish to...

  • inquire about if there are any other details avaiable. For instance, if the sending computer/device placed the E-Mail message in a folder named “Outbox” instead of a folder named “Sent” (or “Sent Items”), that indicates there was an issue with that device/computer getting the message sent to an E-Mail server, and so no later E-Mail server is likely to have any details about that message. In some other cases, a sender might receive a “delivery status notification” (“DNS”) (which might be a “non-delivery report” (“NDR”)).
  • In many cases, issues have been known to be resolved by making sure that an E-Mail server is complying with some of standards (besides just SMTP) which have become popular, such as having a “Reverse DNS” entry (ideally with the phrase “mail.” at the start of the name, and not including certain blacklisted terms like “dyn”), and SPF (and maybe DKIM) (and maybe even also DMARC). If there seems to be trouble with no apparent other cause, implementing those standards is recommended.
  • Checking for unwanted presense on a popular “Registered Black List” may also be prudent. (Determine the IP address of the sending E-Mail server. Then, one may wish to check MXToolbox.com section titled “Blacklists” and/or Anti-abuse.org. The latter site can get a report using a URL that cntains the IP address to look up. Wikipedia's Comparison of DNS blacklists may contain some more information about some DNS RBLs (registerd blacklists).)
Understanding details

Even if you are able to show the details that the Exim software(-generated logs) may provide, understanding what is happening can take a bit of interpretation. (This may require a bit of time from someone who hasn't done this yet.)

An example of incoming E-Mail:

YYYY-MM-DD hh:mm:ss Exim1M-essage-ID H=mail-dm6nam11on1234.outbound.protection.outlook.com (NAM11-DM6-obe.outbound.protection.outlook.com) [40.107.223.68]:36238 Warning: "SpamAssassin as username detected message as spam (6.3)"
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID H=mail-dm6nam11on1234.outbound.protection.outlook.com (NAM11-DM6-obe.outbound.protection.outlook.com) [40.107.223.68]:36238 Warning: Message has been scanned: no virus or other harmful content was found
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID <= sender[\@\.]example.net H=mail-dm6nam11on1234.outbound.protection.outlook.com (NAM11-DM6-obe.outbound.protection.outlook.com) [40.107.223.68]:36238 P=esmtps X=TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=no S=10824 id=AB1CD23EF4567G8HIJ91K2LM34N567891P2345@AB1CD23EF4567.namprd16.prod.outlook.com T="" for whoisto[\@\\.]example\\.org
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID => /home/username/mail/example.net/folder/.Junk/ <whoisto@example\\.org> SRS=<SRS0=12ab3C=TM=example.net=whoisto@example.org> R=virtual_user_filter T=address_directory
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID Completed
  • From Exim's perspective, all E-Mail is incoming (coming into the Exim software) and Outgoing (leaving the Exim software). The reason this message was considered to be an “incoming” message is because the domain name to the left of the < is a local domain. The domain name to the right of the < is an external domain. (If those were flipped, this would be an outgoing/outbound message. If both were local domains, this would be an internal message. If neither were local domains, the E-Mail should not be going through here, presuming that this E-Mail is not set to relay such a message.)
  • The “T=""” contains the E-Mail subject.
  • We can see what happened to the message by looking for one or more lines with =>. In this case, it went into a file. That file is in a mail-related folder. This message went into a user's mailbox named “Junk”. (It is a common representation for the “maildir++” format to have a local directory/folder name start with a period, and then be followed by the visible name of the mailbox folder.) In this less-common case, it looks like the E-Mail message went into a sub-folder named “Junk” which is located in a folder named “folder”.
    • It could be useful to look for a message in that user's folder. The most reliable way to find the message may be to use grep -iR to search for the Exim1M-essage-ID and search throughout the entire user's mailbox.

First part of an example of an outgoing E-Mail:

YYYY-MM-DD hh:mm:ss Exim1M-essage-ID <= sender@example.org H=c-198-51-100-22.hsd1.xy.comcast.net ([10.0.0.181]) [198.51.100.22]:38518 P=esmtpsa X=TLS1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no A=dovecot_plain:sender@example.org S=121728 T="SampleSubject" for getter@example.net
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID Sender identification U=username D=example.org S=sender@example.org
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID SMTP connection outbound 18642034689 Exim1M-essage-ID example.org whoisto@example.net

For the next part of the example outgoing E-Mail, it may very a bit more based on who the message is sent to, or what the resulits of the message is. Here are some examples:

Some details of an outgoing message to an address at GMail:


YYYY-MM-DD hh:mm:ss Exim1M-essage-ID => example@gmail.com R=lookuphost T=remote_smtp H=gmail-smtp-in.l.google.com [74.125.195.27] X=TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=yes C="250 2.0.0 OK  1234567891 23-45678a912b3c456789d12345ef6789a1si12345678pgn.123 - gsmtp"
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID Completed

The above is one example, using gmail.com. Here is a similar example of the last couple of lines when sending to outlook.com.

YYYY-MM-DD hh:mm:ss Exim1M-essage-ID => whoisto@example.net R=lookuphost T=remote_smtp H=example-org.mail.protection.outlook.com [203.0.113,178] X=TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=yes C="250 2.6.0 <a12bcd34-5678-9efa-1234-5b678c9de1fa@MW2NAM04FT018.eop-NAM04.prod.protection.outlook.com> [InternalId=12345678912345, Hostname=MN2PR18MB3181.namprd18.prod.outlook.com] 1234567 bytes in 0.705, 2304.682 KB/sec Queued mail for delivery"
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID Completed
  • The most important part to look at here is probably the three digit number (followed by a space following a “C="”, such as “C="250 ” seen on the last line of the above example. That is a message that is coming from the remote Host (seen in the “Hostname=” field).
    • RFC 5321 section 4.2.1 shows that three-digit numbers starting with a 2 indicate “Postiive Completion” (success).
      • In particular, 250 will be common, which RFC 4.2.3 indicates means “Requested mail action okay, completed”.
      • If you see a 250 result, then check what Host provided that information (by checking the “Hostname=” field).
        • If the local E-Mail server simply relayed the message to another local server supported by the same technical support staff, then the next troubleshooting step may involve looking at information on that relaying computer, and seeing what happened after the message got relayed to that next hop. Checking a local system, which is supported by the same staff, would be a step most sensibly taken before contacting the recipient.
        • In another (probably more likely) scenario, if that is a remote system more related to providing service for the recipient (rather than the sender), then further troubleshooting will likely need to be done by the recipient (or the recipient's IT/technical support). It may be sensible to look for an automated reply from that domain (possibly providing an NDR/DSN (non-delivery report/delivery status notification) notice), but otherwise it is typically best to check with the recipient.
          • The comment on the 250 line (which is the text showing after the “250 ” may include a message ID which is assigned by the remote E-Mail software. Having that available may be helpful for the remote system's support to find relevant information on the E-Mail message delivery attempt.
    • A number in the 400-499 range may indicate a “transient” status (temporary result), and “the SMTP client SHOULD try again.” according to RFC 5321 section 4.2.1.
      • While 450 (which RFC 5321 page 51 calls “command rejected for policy reasons”) might have been sensible for grey-listing, 451 (which RFC 5321 page 51 calls “Requested action aborted: error in processing”) is apparently a more common response for greylisted messages.
    • RFC 5321 section 4.2.1 indicates that a number in the 500-599 range may indicate “permanent negative completion”. In other words, the results of a specific attempt was rejection. (RFC 5321 section 3.3 notes, “Normally, failures produce 550 or 553 replies.”)

Occasionally, you might not see a “ -> ” followed by a “ C="”. Another example of an outgoing E-Mail, where the message was not successfully relayed to the remote host, is shown below:

YYYY-MM-DD hh:mm:ss Exim1M-essage-ID <= sender@example.org H=c-198-51-100-22.hsd1.xy.comcast.net ([10.0.0.181]) [198.51.100.22]:38518 P=esmtpsa X=TLS1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no A=dovecot_plain:sender@example.org S=121728 T="SampleSubject" for getter@gmail.com
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID Sender identification U=username D=example.org S=sender@example.org
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID SMTP connection outbound 18642034689 Exim1M-essage-ID example.org sender@example.net
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID ** whoisto@example.net R=lookuphost T=remote_smtp H=gmail-smtp-in.l.google.com [74.125.195.27] X=TLS1.2:ECDHE-ECDSA-AES128-GCM-SHA256:128 CV=yes: SMTP error from remote mail server after end of data: 550-5.7.26 This message does not have authentication information or fails to\n550-5.7.26 pass authentication checks. To best protect our users from spam, the\n550-5.7.26 message has been blocked. Please visit\n550-5.7.26 https://support.google.com/mail/answer/81126#authentication for more\n550 5.7.26 information. n18-20020a170902d2d200b0014f9c665c16si11156931plc.213 - gsmtp
YYYY-MM-DD hh:mm:ss Exim1M-essage-ID Completed
  • This indicates that the remote server had an issue with the E-Mail message.
  • This specific example involves sending an E-Mail message to an address @gmail.com. However, don't just pass on troubleshooting to the receiving computer (which, in this case, is Google). If you try to have an end user contact Google, that is likely to only end in frustration because the Internet giant is likely to provide almost no information except for referring to the web page that the E-Mail logs already specified. Instead, try to figure out why the Internet giant known as Google is refusing the E-Mail, and see if cooperative steps lead to the message being accepted. (The “end user” sender is likley to appreciate this approach so much more.) Try the following...
  • The above sample text is nearly what was seen in an actual incident. Here are the steps that resolved the case when the receiving computers complained that “authentication information” was not sufficiently passing the “authentication checks”.
    • This is an error that can commonly happen with an E-Mail server that does not have corresponding authentication checks set up in DNS. You may wish to verify this by lookin up TXT records. e.g.:
      • nslookup -q=TXT example.org
    • See if there is a valid SPF record. If so, you might also wish to check to make sure there is valid DKIM and DMARC set up.
      • If things are not set up well, then:
        • verify what computers are acting as the DNS nameserver (e.g. nslookup -q=NS example.org )
        • Then, ensure the relevant authoritative DNS server(s) get updated. Then make sure that Google's DNS servers see the update. e.g.:
          • nslookup -q=TXT example.org dns.google
          • Once the right record(s) are in place, try again (by having the sender make another attempt to re-send the E-Mail message).
[#eximhndq]: Handling a queue

This section contains some commands that may be found elsewhere, and some custom variations. After a small number of experiences, some of the data (after the introductory text) was re-ordered so that some of the most useful information gets seen (and possibly acted on) earlier.

There might be multiple reasons to interact with a queue. At the time of this writing, this guide may still be somewhat heavily focused/slanted towards addressing situations where an outbreak of outgoing spam has led to messages being in the queue.

Things to do

This guide might not be fully done. Some items known to remain, but which just hadn't yet been reached, include:

...

  • section on freezing/deleting the full server is sparse. (Maybe because info is in a simlar section elsewhere?) (e.g., maybe it is just sparse for Mailer Daemon, but not for high-sending user? Or similar/etc.?)
  • A general review/clean-up, including removing some older material at the end which is expected to just be sloppy/redundant
  • That's it! (The other items on this task list have been addressed.)
  • There may yet be some untested material here at the moment... This page had a lot of commands re-designed since the last significant test. The re-designs result in searching through messages less, so the processes should go much faster (but use temporary files and multiple commands, instead of longer commands that rely on even more piping and not re-using prior results found in temporary files). Much of this isn't tested well yet, although it is a modification from some commands that were useful. (So if anything doesn't work, hopefully it won't be far from being useful, and can be fixed when this gets checked again next time.)
Intro
  • Naturally, always consider the impact of each action before proceeding. Far too many people have been known to just empty out a queue, potentially eliminating valid messages when a different, smarter approach might have been able to save such valid data.
    • System administrators are encouraged to take responsibility for their actions. For instance, this guide has people storing things in /tmp/. Some operating system(s), including OpenBSD, maybe known to delete the contents of /tmp/ during a reboot. If you are using an operating system that does this, and think you might care about such files, it might be more prudent to store them somewhere underneath /var/. Actually, the files that this guide puts in /tmp/ are probably not critical in the long term, particularly if the most useful details are documented elsewhere (like a ticketing system), but this is just a reminder to familarize yourself with your system's behavior and make intelligent decisions that you will be responsible for, instead of just blindly trusting and applying details found from a resource like this guide.

This guide makes use of some intermediate files. Some of that is rather unnecessary: multiple commands could be strung together to create a longer command with more piping and less temporary files made along the way. The reason why this guide decided to use temporary files is that seemed to have potential to save time, especially when an incident may involve a larger number of messages that may need to get processed. Time was able to be saved by eliminating the need to process some of that data multiple times.

(This guide does presume there is enough free space for such temporary fils, which have been known to be tens of megabytes (at a local ISP which coud have tens of thousands of messages, once having about 176,000 messages still in the queue). Naturally, the amount of disk space required is likely to vary based on system usage, just as the amount of time may vary based on how much data is being processed and what hardware is being used.)

[#eximqvrf]: Verify issue
  1. Get message count
    • exim -bpc
      • This has potential for taking a while - even a few or several minutes. Usually it will go pretty fast (seconds) if there isn't a problem. You may want to run: time exim -bpc
      • Typically want low-- zero or near zero-- maybe a dozen at the most.
      • However, that isn't always true. For instance, at one local ISP with eight cPanel servers, there was one of them that might have hundreds on a normal basis. So, having a pre-documented benchmark can be helpful to determine if the machine is acting rather abnormal, according to what is normal for that machine.
[#eximqckc]: Check Cause
  • See if system load has been an issue:
    grep -i "no immediate delivery: load average" /var/log/exim_mainlog | wc -l | ${SHELL} -c "time -p cat"
    gzip -dc $( ls -1tr /var/log/exim_mainlog?2*.gz ) | grep -i "no immediate delivery: load average" /var/log/exim_mainlog | wc -l | ${SHELL} -c "time -p cat"
    • That parameter to ls is: hyphen, one (not lowercase L!), t, r
    • The usage of the time command is optional and may seem pointless, particularly if the turnaround time is fast. (What it basically does is lets you know how long you needed to wait for a response, which has been known to exceed 30 seconds in some cases.) The main number we are going to be interested in is the first line.
    • Ideal number for the first line of output: zero.
    • If above that ideal number, the server was busy. Perhaps that was due to spam. To check for other potential causes, consider taking a look at a guide on Unix: Slow Systems. That guide may also be helpful if you have a high queue because the server load is too high for some reason/cause other than E-Mail.
    • If an issue, you may wish to find out more details about what the system was doing at that time. On some systems, sar may be helpful. See: Slowness on Unix-based Platforms: “Prior System Slowness”.
      • Since slowness can be different then system load, it may be worthwhile to re-configure (or disable?) this behavior. If Exim's configuration does enable this behavior of shutting off outbound mail when the load is too high, then this behavior can be configured to determine what load level this happens with.
[#eximqdso]: Disabling Outbound Mail

This might or might not be a good idea.

  • On the positive side, turning off service for the entire server may be an action that can be taken more quickly than assessing other data first. Functionality can then be restored after things are in better shape, and the impact on legitimate E-Mail messages can be nothing more than a rather minimal delay (of maybe a few minutes, mostly while things are being investigated.)
  • On the disfavorable side, this may affect legitimate E-Mail messages and be more work overall. Ignoring this step may cause later steps to be accomplished sooner. If a more targeted approach can be resolved in nearly the same amount of time, then the extra hassle of impacts related to taking this step might be deemed to not be worthwhile effort/time to spend.

If you do wish to proceed with this, then...

  • (At this time, consider this particular step to be less well-tested than some of the later steps. Also, information here may be a bit more on the sparse side, compared to detailed steps shown later in the guide.)
  • Later steps describe how to identify which E-Mail address(es) sent many E-Mail messages, and how to disable the abilities for just such address(es) to send E-Mail. However, if you really want to absolutely minimize outbound spam, you might wish to completely disable outbound E-Mail while other steps are taken. (How useful this is might depend on how quickly some later steps can get done. Note that remote/public blacklisting decisions might be made, at least in part, by how much spam is being sent out. So minimizing outgoing spam may be desirable for selfish/self-centered/self-focused reasons as well as more altruistic concerns.)
  • LowEndTalk.com: stop outgoing remote emails, and allowed incoming mail (exim) discusses some methods. Here is one method, based on the idea mentioned by a post by alshahad:
    • NOTE: This has not been verified by the creator of this documentation as of the time of this writing...
    • Back up a file that you plan to change...
      pwd
       cd /etc/
       echo cp /etc/exim.conf{,$( echo mktemp /etc/exim-backup-$( date +%Y%m%d%H%M%S%z )-XXXXXXXXXX.conf )}
       \cp -v /etc/exim.conf $( mktemp /etc/exim-backup-$( date +%Y%m%d%H%M%S%z )-XXXXXXXXXX.conf )
      • The \ before cp is to prevent a common alias of using cp -i from prompting about overwriting a file created by “mktemp”.
      • The echo command is pointless, and only left here for reference so information isn't being removed too quickly/carelessly. It is likely that a futrue version of this guide may remove that (and hopefully this note as well).
      • This sample command may be a bit tricky for newcomers and maybe even some experienced technicians, as it uses shell features at least three times: the feature of using a “command substitution” (using the “$(...)” is used twice, nested, and even that layer of complexible is stored in a “brace expansion”-style wildcard. None of this will work if a shell does not support these features.
        • The idea behind the brace expansion is that sh will convert the braces and contents into “/etc/exim.conf” followed immediately by whatever came between the opening curly brace and the first comma, followed by a space, and then “/etc/exim.conf” followed immediately by whatever came between the opening curly brace and the second comma and the closing curly brace.
    • Restart Exim.
      • This suggestion was added in response to a situation where an older server didn't update ClamAV okay. Then, when restarting Exim, an error message resulted. It might be nice to restart Exim before making changes, so that if any error messages arise, you don't need to question if those changes caused this. (This might not be worthwhile if it causes a little bit of downtime is a higher priority than a technician's earlier confidence that things are working as well as they are going to. Like other advice on this page, this is just a recommendation and might be something that some people disagree with and choose not to do, which may be just fine.)
    • Look for two lines in the file:
      remote_smtp:
       driver = smtp
    • After that, add a line (presuming that it doesn't yet exist)...
      remote_smtp:
       driver = smtp
       interface = 203.0.113.25
      • You can use the text 203.0.113.25 literally. The goal here is to specify an invalid address, and that address should not be getting used for public communication as noted by IETF RFC 5735 page 5.

    • Have Exim reload its configuration file.

      [#eximrecf]: Reloading Exim Configuration
      • According to BradTheMad.org's “Exim Cheatsheet”, this can be done using SIGHUP. Try:
        kill -HUP $( cat /var/spool/exim/exim-daemon.pid )
      • The other option is to restart Exim:
        [#eximrego]: Restarting Exim

        Before restarting Exim, determine why. Specifically, there is some advice to not induce unnecessary downtime if your real goal is just to be Reloading Exim Configuration.

        How this is done may vary between operating systems. Some examples could include: “ service exim restart ” (or “ service exim start ” may be appropriate if the Exim software is known to be stopped), or “ /etc/init.d/exim restart ” Machines using the suite known as cPanel might have a “ /scripts/restartsrv_exim ” command.

Prepare for clean-up

You may wish to start out by running:

ls -l /tmp/exim{bp,grp}*

If there are any pre-existing files, you may wish to figure out how you would like to handle them.

  • They can be ignored. It's just that if you do, then new data might be somewhat merged with older data. That isn't expected to cause any issues if you limit yourself strictly to the commands (in order) shown in this document, but otherwise could result in a situation that could potentially be a bit more confusing.
  • If they are dated today, you might want to see if any other authorized administrators might already be working on the same server (following this guide or some other steps based on a variation of this guide).
  • You may wish to move old stuff out of the way (possibly moving it to oblivion, a.k.a. deleting it) before proceeding to make new data.
Quick Summary Of Upcoming Steps

You may want to see:

[#eximqbgs]: Determine Biggest Sender(s)

Show biggest senders of un-delivered messages, with one notable caveat:

  • This technique may be just for E-Mail messages not including the local “Mailer-Daemon” account, which will be checked later.

  • Create the data file(s) used by more commands.

    [#gteximbp]: Useful Data From Exim

    Use the following directions to first create the /tmp/excimbp.* file, and then to use that file to help create the /tmp/excimbp.* file.

    Creating the /tmp/excimbp.* file file
    New Version

    This is untested at the time of this writing, but probably displays nicer (at least on some operating systems) better than the old version mentioned next:

    date ; time exim -bpr | tee $( mktemp -p /tmp eximbp.XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat" ; date

    (This was created noticing that the old version resulted in CloudLinux's AlmaLinux OS just displayed time right away.)

    Old Version
    exim -bpr | tee $( mktemp -p /tmp eximbp.XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
    • This guide uses a data file, because once the information gets extracted from exim (or similar commands that extract data from exiqgrep), the same data file can be checked for details more quickly. When there are a large number of messages in the queue, this can often go much more quickly.
    • An earlier version of this guide ran “exim -bp” instead of “exim -bpr”. A lot of other guides seem to utilize “exim -bp”. However, that can be notably time-consuming when there are many messages in the queue. Apparently “exim -bpr” can be notably faster. The difference is that “exim -bp” sorts results chronologically. While that can be nice, it isn't very necessary for many cases, and might be causing a noticeably slower result.
    Creating the /tmp/excimbpid.* file file
    New Version

    This is untested at the time of this writing, but probably displays nicer (at least on some operating systems) better than the old version mentioned next:

    date ; time cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep -vE "^ {8}.*" | grep -v "^$" | awk "{print \$3}" | tee $( mktemp -p /tmp eximbpid.XXXXXXXXXX ) >> /dev/null ; date

    (This was created noticing that the old version resulted in CloudLinux's AlmaLinux OS just displayed time right away.)

    Old Version
    cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep -vE "^ {8}.*" | grep -v "^$" | awk "{print \$3}" | tee $( mktemp -p /tmp eximbpid.XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"

  • See the sender for E-Mail messages, and which outgoing addresses each copy is trying to be sent to:

    • Note: Upon generating the results of the next command, it is recommended that you may not want to spend a lot of time studying the data related to the problem before moving on at least one more step. That will often help identify some of the biggest senders. Then, with that additional information in mind, if you want to re-run the second cat command in this section in order to study things a bit closer, that may make sense to do then.
    cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep @ | cut -d \< -f 2 | cut -d \> -f 1 | tee $( mktemp -p /tmp eximgrp.highsendrecip-XXXXXXXXXX ) >> /dev/null | wc -l | ${SHELL} -c "time -p cat"
    ( cat $( ls -1t /tmp/eximgrp.highsendrecip-* | head -1 ) && echo Sender + Related Receivers List: Data @ $( ls -1t /tmp/eximgrp.highsendrecip-* | head -1 )) | ${SHELL} -c "time -p cat" | less
    • The parameter to ls is a one, not a lowercase L
    • The first command in that pair is used to generate data used by the second command in that pair, and also used by later commands discussed here.
    • The second command shows information, and is optional if you don't feel a need to see that information. However, if you do find it worthwhile, you may wish to make a look at the filename (shown in the last line of output).

    See how many times standard E-Mail addresses have been sending a message.

    cat $( ls -1t /tmp/eximgrp.highsendrecip-* | head -1 ) | grep -Ev "^ {8}. " | sort | uniq -c | sort -nr | tee $( mktemp -p /tmp eximgrp.highsendat-XXXXXXXXXX ) >> /dev/null | wc -l | ${SHELL} -c "time -p cat"
    ( cat $( ls -1t /tmp/eximgrp.highsendat-* | head -1 ) && echo Sorted Sent Most Pre-Forked: File data @ $( ls -1t /tmp/eximgrp.highsendat-* | head -1 ) ) | ${SHELL} -c "time -p cat" | less
    • Seeing who is sending most of the E-Mail messages can often be done rather easily by running (or re-running) that last command.
      • (This is often very useful. You may wish to make note of the name of the data file, so you can easily view that directly in case repeating this command doesn't work as well (which can happen if only some steps get repeated later on).
    • Command Details:
      • The parameter to ls is a one, not a lowercase L
      • The parameter seen in some grep commands is: a quotation mark, then a carrot, then ten (10) spaces before the next quotation mark.
    • Results commentary:
      • Note: This likely under-reports severity, but is useful regardless. For instance, if an E-Mail address tried to send out one message to 12 different recipients, and then another message to 8 different recipients, and then another message to 10 recipients, then this looks like 3 messages, even though there are 30 recipients. Fortunately, even this style of counting is often sufficient to start to notice what E-Mail address is sending to many different recipients.

  • See how many E-Mail messages are in the queue to be sent from the server's local Mailer-Daemon E-Mail address.

    cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep \<\> | cut -d \< -f 2 | cut -d \> -f 1 | grep -Ev "^ {8}. " | uniq -c | ${SHELL} -c "time -p cat"
    • The parameter to ls is a one, not a lowercase L
    • The parameter seen in some grep commands is: a quotation mark, then a carrot, then ten (10) spaces before the next quotation mark.

  • See a number of messages that have probably already been sent out as spam.

    New version
    date ; cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep @ | cut -d \< -f 2 | cut -d \> -f 1 | grep -Ei "^ {7}.D " | tee $( mktemp -p /tmp eximgrp.partsent-XXXXXXXXXX ) >> /dev/null ; date
    wc -l $( ls -1t /tmp/eximgrp.partsent-* | head -1 ) | ${SHELL} -c "time -p cat"
    Old version
    cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep @ | cut -d \< -f 2 | cut -d \> -f 1 | grep -Ei "^ {7}.D " | tee $( mktemp -p /tmp eximgrp.partsent-XXXXXXXXXX ) >> /dev/null | wc -l | ${SHELL} -c "time -p cat"
    wc -l $( ls -1t /tmp/eximgrp.partsent-* | head -1 ) | ${SHELL} -c "time -p cat"
    • The parameter to ls is a one, not a lowercase L
    • The parameter seen in some grep commands is: a quotation mark (which the shell processes before passing it onto grep, then a carrot, then seven (7) spaces, and then a period and an asterisk, and then the capital letter D, and then one more space before next quotation mark.
      • The periods are there because there might be a “+D” after the 5 spaces, or there might be 2 spaces there. Then, there will be one more space.there may be an eighth space (probably more common) or “+D”. Or maybe it is an eighth space and then “+D”. (This was based off of Unofficial man page for exim(8), made for Debian GNU/Linux, located at linux.die.net (on a site named after the singular version of the more famous plural word, dice) “-bpa” mnetioning +D, so, since it wasn't seen yet, this command is currently based on documentation and not a verified instance.)

    The number of messages shown here is the number of E-Mail addresses that have received delivery as part of an E-Mail message that only got partially delivered because other E-Mail messages have not had the message delivered. (While there might be some non-spam in that list, if there is quite a bit of spam then the number of legitimate messages in this state may likely be a relatively small amount. E-Mail messages that had all of the recipients delivered won't be part of this count, so if anything, the actual number of spam messages sent may likely be even higher than this number.

[#eximqbsm]: Some Basic Overview Summaries

This section describes some additional summary reports that may also be nice to overview.

  • Many of these may interact with data that was presumably created earlier, if this guide was processed in order. If you see a command that references /tmp/eximbp.* or /tmp/eximbpid.* files, then that command likely won't work well unless those files already exist. If you haven't made those files yet, see the section titled “Useful Data from Exim” (which is within a larger section named Determine Biggest Sender(s)).
  • Unnecessary, but nice:
    • These might not be needed. However, they can help to verify the existance of some problems, and may be helpful to identify problems when other approaches seem complete.
    • It is recommended to check these out, and see what kind of numbers show up. But then, follow the steps described later about identifying which E-Mail address(es) have sent the most stuck messages, and clean up those messages. When cleaning up messages with those directions, problems identified by these commands are likely to get reduced (hopefully down to non-existence).
    • (In a super critical emergency where urgency outweighs gathering of information which may be helpful immediately or over a longer term, these summaries could be skipped.)
count frozen

See how many messages are frozen:

cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep frozen | wc -l | ${SHELL} -c "time -p cat"
  • (If a command fails because the /tmp/eximbp.* or /tmp/eximbpid.* files don't pre-exist, see the section titled “Useful Data from Exim” (which is within a larger section named Determine Biggest Sender(s)) (as noted in the section titled HREF="#eximqbsm">Some Basic Overview Summaries).
  • In most typical cases, the desired amount is typicaly low-- zero or near zero-- maybe a dozen at the most.
    • (A high number of frozen messages might indicate a high number of messages that have been having some troubles getting processed/delivered.)

If that is a relatively small number (e.g., fewer than 20 messages), you might wish to capture a list of what is pre-frozen. That way, you may later decide to freeze more of them, with intent to unfreeze messages later, but will remember which are worth freezing later. For example:

cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep frozen | ${SHELL} -c "cat"

Some current speculation...

  • One typical result seems to be that there are no pre-frozen messages. However, that does not seem to always be true.
  • Perhaps a typical result is that before anything gets manually frozen, all pre-existing frozen messages might often fit the pattern of having these three strings:
    • "F From: Mail Delivery System "
    • " Subject: Mail delivery failed: returning message to sender"
    • "Message-Id: <.......-......-..@$(hostname)>
    (except, naturally, having the actual host name)
  • Note: some of the reports may insert some ANSI sequences.
  • It might be interesting to test for these cases. In such cases, a standard reaction might be appropriate. (Just deleting such messages might make sense.)
exiqsumm's summary

See exiqsumm's summary:

cat $( ls -1t /tmp/eximbp.* | head -1 ) | exiqsumm | sort -bgr | tee -a $( echo mktemp /tmp/exim-backup-$( date +%Y%m%d%H%M%S%z )-XXXXXXXXXX.conf ) | ${SHELL} -c "time -p cat" | less
  • (If a command fails because the /tmp/eximbp.* files don't pre-exist, see the section titled “Useful Data from Exim” (which is within a larger section named Determine Biggest Sender(s)) (as noted in the section titled HREF="#eximqbsm">Some Basic Overview Summaries).
  • This summarizes messages, and can be a good way to identify some major trends like destination domains (If you want to see how well things have cleared up after some prior action, “ exim -bpc ”, may be a better indicator for a simple count showing the remaining queue size.
  • In the above example, the output from time has been known to overwrite output from less. (Maybe that is a race condition and inconsistent?) However, once less overwrites the screen (with Ctrl-L or arrow keys), that may overwrite the screen and end up effectivey permanently erasing visbility of the information that the time command output.
TLD sort

See which specific TLDs (top-level domain names) that show up most in addresses of messages stuck in the queue...

First, make sure you have the “eximkeyheaders file. (If you are following this guide in sequence, then you probably don't have that yet, so follow the next steps on creating it.)

[#gtkyhdin]: Getting “Key Header Info” For Processing

The key command to do this may take a while. The time is expected to vary between systems. As an example, in 2022 a real live system took about 85 seconds to process about 26.888 E-Mail messages per second. So after running “exim -bpc”, which is often quick, you may then be able to calculate how many minutes you might want to break/walk away for.)

  1. (If a command fails because the /tmp/eximbp.* or /tmp/eximbpid.* files don't pre-exist, see the section titled “Useful Data from Exim” (which is within a larger section named Determine Biggest Sender(s)) (as noted in the section titled HREF="#eximqbsm">Some Basic Overview Summaries).
  2. Check if needed
    • Since this command may take a while, you may wish to see whether you need to do this. Run:
      ls -l /tmp/eximkeyheaders.*-??????????
       echo Value of MSGPATID is =${MSGPATID}
       ls -ltr /tmp/eximkeyheaders.${MSGPATID}-??????????
      • Note: This stage of the process may not require a specific value for the ${MSGPATID} variable.
        • To rephrase that in a single sentence: For this stage of the process, It may not matter much what ${MSGPATID} is set to. It could even be unset (effectively equal to blank/null), and that may still work. However, if some other commands also use the ${MSGPATID} variable then that variable may need to be used with a value that is consistent in order to have commoands successfully interacting with the data created by the correct other commands.

      If one or more files showed up with that last command, then look at the most recent file and determine whether that is new-enough data. (For instance, if you had an incident a few months back and never deleted the log files, then that data is probably outdated and you should get some updated data. If the data was made in the last 15 minutes, maybe you don't want to waste multiple minutes trying to get an update that is newer than what is already available.)

  3. If (new) data does need to be created (in a /tmp/eximkeyheaders.${MSGPATID}-?????????? file), then:
    • Make sure there is recent eximbpid data.
    • Run this:
      date ; cat $( ls -1t /tmp/eximbpid.* | head -1 ) | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}"; exim -Mvh "${arg}" ; done ' | tee $( mktemp -p /tmp eximkeyheaders.${MSGPATID}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat" ; date

(The process of creating the above data file may be doing a bit more work than what is needed for the TLD sort. However, by doing that, the same data file can be re-used, eliminating a longer wait period used by a command described later in this guide.)

Then, the next command points to the data file created in a previous command.

cat $( ls -1t /tmp/eximkeyheaders.${MSGPATID}-* | head -1 ) | grep -i "To: " | cut -d " " -f 3-99 | cut -d \< -f 2 | cut -d \> -f 1 | grep -v @cp.*\\.example\\.com | rev | grep -i \\. | cut -d . -f 1 | rev | sort | uniq -c | sort -bgr | grep -v "[0-9] com" | grep -v "[0-9] org" | grep -v "[0-9] net" | ${SHELL} -c "time -p cat"
  • The one part needing customizing is where the above example shows “@cp.*\\.example\\.com”. This example came from an environment where there were multiple authorized internal E-Mail servers that had names that fit the pattern like “cp###.example.com”, where “example.com” represents a domain name which was owned by the service provider (not a customer domain). This example was used to demonstrate how to make such an excusion. If you don't need such an exclusion, especially if that seems non-intuitive, then feel free to just leave out that particular portion (starting with the “grep just before that text, and going up through the redirection pipe (“|”) afterwards.
  • This doesn't report on the messages going to “.com” or “.org” or “.net” (only because of the “ grep -v "[0-9]"” commands at the end).

Effect:

  • This summarizes messages, and can be a good way to identify some major trends like destination domains (If you want to see how well things have cleared up after some prior action, “ exim -bpc ”, may be a better indicator for a simple count showing the remaining queue size.)

(Because of the earlier command that created the eximkeyheaders/*-* file, the first command in the next section uses that earlier data file. That approach goes much faster than if that earlier data file doesn't pre-exist and needs to be created.)

date ; cat $( ls -1t /tmp/eximbpid.* | head -1 ) | xargs -L 1 echo A command to show headers could be: exim -Mvh | tee $( mktemp -p /tmp eximbpheaders-tmp-for-count.fortldsort-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat" ; date
wc -l $( ls -1t /tmp/eximbpheaders-tmp-for-count.fortldsort-* | head -1 )
[#eximqdpl]: Deeper Looks
  • These might be NOT needed.
    • Especially when there is actual problems occuring (like spam being sent through the server, or downtime of functionality that is effectively broken due to inclusion on a DNS-based E-Mail registered blacklist), there may be no need for these “deeper looks”. However, if you are wanting to see what is happening in the queue, some of the details here may be helpful to keep in mind.
  • Overview of this section:
    • These are commands that may be run when a technician is trying to figure out some details of what the spam's content looks like (including logged details of the connection, message headers, and message body). These commands are not necessarily needed, and may often be skipped so progress can be obtained quicker with some of the later commands. However, if there are questions about the contents of the spam, then knowing about these may be beneficial.
Summaries of data from all msssages
Summarized Key Headers (All Messages)

This may help to quickly provide an enlightened view by seeing some key details of all messages remaining in the queue.

  • Summary: This shows summarized headers of (all) messages in the queue. (This is similar to the later commands showing summarized headers of just messages fitting certain patterns like having a certain sender or certain top-level domain(s) in the list of recipients...)
  • Comparison: This shows less details than the following section that describes how to see the entire messages. Instead, this restricts output to some of the more useful items to check. By including fewer lines from a message, such key information from multiple messages can be seen more quickly and easily.
    • (This was initially found to be very enlightening, although this guide now has some similar commands that provide some more specific details that may be narrowed down more in ways that are even more useful. Still, this is provided in case a more general overview is desired.)
Gather data if needed

Note: If you did the “TLD Sort” above, then you already created the file that gets made by the next command, so feel free to skip this next, upcoming command (and then proceed with the following instructions after this upcoming command).

(Updated process: instead of running the next command, go through the sub-process described by the section titled Getting “Key Header Info” For Processing, as going so helps determine whether a step is needed, which could save time compared to unnecessarily performing a rather time-consuming step. Once this guide's maintainer carefully ensures the following command does exactly match some information from that other section, it may be getting removed from here...)

cat $( ls -1t /tmp/eximbpid.* | head -1 ) | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}"; exim -Mvh "${arg}" ; done ' | tee $( mktemp -p /tmp eximkeyheaders.${DISABLED/WAS/MSGPATID}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
  • (If a command fails because the /tmp/eximbp.* or /tmp/eximbpid.* files don't pre-exist, see the section titled “Useful Data from Exim” (which is within a larger section named Determine Biggest Sender(s)) (as noted in the section titled HREF="#eximqbsm">Some Basic Overview Summaries).

cat $( ls -1t /tmp/eximkeyheaders.${MSGPATID}-* | head -1 ) | nl | grep --color=always -Ei "\^\$"\|"====="\|" {2}-host_address "\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: "\|" X-Spam-Status: Yes, score="\|" X-Spam-Flag: " | tee $( mktemp -p /tmp eximrep-keyheaders-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
less -R $( ls -1t /tmp/eximrep-keyheaders-* | head -1 )
Seeing Delivery Status

The command for this is one of the simpler commands that makes use of some data gathered earlier:

cat $( ls -1t /tmp/eximbp.* | head -1 ) | less
  • (If a command fails because the /tmp/eximbp.* or /tmp/eximbpid.* files don't pre-exist, see the section titled “Useful Data from Exim” (which is within a larger section named Determine Biggest Sender(s)) (as noted in the section titled HREF="#eximqbsm">Some Basic Overview Summaries).

However, a bit of help interpreting may be in order...

  • The following is a brief overview of what the information of the lines is believed to mean...
    ##h #.#s MsgId1-MsgID2-xx <?sender@example.com>
    ?recipient-of-frozen-mail@example.net
    D ?recipient-of-recently-delivered-mail@example.net
    • The first bit of time is how long ago the initial attempt was to deliver the message.
    • The next section indicates the size.
    • The third section (shown as “MsgId1-MsgID2-xx” in the above example) represent a message ID
      • The part between the < and the > represent who the E-Mail address is from.
      • That part between the < and the > might be blank (so it just looks like “<>”). Such a blank E-Mail address is actually short-hand for Mailer-Daemon@ (the local computer).
  • This may show both delivered messages and not-delivered messages. (If a single E-Mail message is sent to multiple recipients, this may show the messages which are delivered copies, as well as showing the messages that are undelivered copies.)
    • To not show the delivered mail, here is a variation:
      cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep -vE "^ {7}.*D " | less
Seeing Details of Individual Messages

To utilize commands in this section, you'll first need to identify the Exim “Message-ID” related to an individual E-Mail message. (There may be various ways of seeing such an Exim “Message-ID”. One is using exiqgrep with the “-i parameter which causes exiqgrep to just show the Exim “Message-ID” for each message that matches (the query that is given to exiqgrep).

Seeing a message's headers
  • If you have the Exim Message-ID for an E-Mail message, you can run something like:
exim -Mvh UseThe-Messag-ID
  • Naturally, the “UseThe-Messag-ID” is meant to be customized. (Instead, use the actual Exim “Message ID” for the individual E-Mail message.)
  • Presumably, the parameters might mean something like “Mail/message data”, “verbose”, “mnessage header information”.
Seeing a message's body
exim -Mvb UseThe-Messag-ID
  • Naturally, the “UseThe-Messag-ID” is meant to be customized. (Instead, use the actual Exim “Message ID” for the individual E-Mail message.)
  • Presumably, the parameters might mean something like “Mail/message data”, “verbose”, “mnessage body data”.
Seeing logged info about a message
exim -Mvl UseThe-Messag-ID
  • Naturally, the “UseThe-Messag-ID” is meant to be customized. (Instead, use the actual Exim “Message ID” for the individual E-Mail message.)
  • Presumably, the parameters might mean something like “Mail/message data”, “view/verbose”, “mnessage logged details”.
[#eximqsnh]: Handle high-sending E-Mail address(es)

This guide starts with looking at a typical E-Mail address, on the assumption that a typical E-Mail address may be compromised and is likely to be involved with creating more trouble than the local server's Mailer-Daemon account. (Handling the Mailer-Daemon account will be looked at more later.)

Handling a spamming E-Mail Address

Identify which E-Mail addresses have been sending a lot of messages. To do this:

Extract info

Extract information for E-Mail that fits a pattern, such as E-Mail that comes form a specific address. For example:

Make a variable...

Make a variable...

MSGPATID=patone
(Might want to create a variable. e.g. MSGPATID=patone
  • (A future version of this guide may do so, after verifying that using such a name in $( ) sub-shells works as anticipated.)

Customzie and Extract
  • (Reminder: customize the E-Mail address in the next line!)

Extract the data:

exiqgrep -i -f \<username@example.org\> | tee $( mktemp -p /tmp eximgrp.${MSGPATID}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
  • (The “${MSGPATID}” simply refers to pattern number one. The “pattern” of messages is what exiqgrep checks for when analyzing an individual message.)
    • (If you wanted to, you could extract data from multiple users, maing slight changes to the filename here and elsewhere...)
Generate and View

Generate and view report for this E-Mail address...

The good way...

First, gather some data.

exim -bpc
cat $( ls -1t /tmp/eximgrp.${MSGPATID}-* | head -1 ) | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}"; exim -Mvh "${arg}" ; done ' | tee $( mktemp -p /tmp eximverbheaders.${MSGPATID}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
  • The reason to run “ exim -bpc ” again is just to provide a reminder of how many messages there are. As of 2022, an actual production system was able to process about 26.888 messages per second, so two minutes could process about 3225 messages in the queue. This may give an indication of how long you might need to wait (or take a break)...

Then, generate a report which looks at the headers related to messages from the specific E-Mail address being processed. (Presumably, if you do this multiple times with different E-Mail addresses, you would have a different value for “${MSGPATID}”.)


cat $( ls -1t /tmp/eximverbheaders.${MSGPATID}-* | head -1 ) | nl | grep --color=always -Ei "\^\$"\|"====="\|" {2}-host_address "\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: "\|" X-Spam-Status: Yes, score="\|"Failed to open input file for .{6}-.{6}-..-H: No such file or directory"\|" X-Spam-Flag: " | tee $( mktemp -p /tmp eximrep.${MSGPATID}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
( cat $( ls -1t /tmp/eximrep.${MSGPATID}-* | head -1 ) && echo MsgPatID -\"${MSGPATID}\"- file data was in $( ls -1t /tmp/eximrep.${MSGPATID}-* | head -1 ) ) | less -R
  • If you like the above data, consider making a note of the filename to easily reference it later. (The output of the less command will show that filename, but that isn't actually part of the contents of the main file.) If there are multiple E-Mail addresses that are getting processed in this way, then this guide also suggests making a note of which E-Mail address that was for. (That can be seen easily by checking the “From:” field.)
  • The above example may seem a bit redundant to have both X-Spam-Flag and X-Spam-Status:. The idea is that if you want to see how spammy things are, you could juse use “X-Spam-Status: Yes, score=”, but if you wanted to quickly check for ham (non-spam) that should be preserved (instead of deleted/removed), you might want to look for “X-Spam-Flag: ” for the report (and separately search for “X-Spam-Flag: NO”
    • That idea is somewhat untested. Maybe it is just as easy to filter based on seeing “X-Spam-Status: ” and then count/search for “X-Spam-Status: No, score=”??
The botched way...

At first glance, this seemed like a faster way. However, this ends up showing data about other senders as well, which is undesirable. (For instance, if the resulting report shows details about other messags, and then information is extracted from that report, and messages deleted, we do NOT want other unrelated messages to get the same treatment!) For now, this idea is left here as a reference of what should NOT be done. (This idea was documented before being thought through fully enough to determine the problem with this faster approach that gives the desired data, but not only the desired data...)

Gather data if needed

Note: If you did the “TLD Sort” and/or the “Summarized Key Headers (All Messages)” steps earlier, or if you did this same command for a different username earlier, then you may already have created the data file that this first step performs. In that case, don't waste time re-running the command that was run before, and which does not need to be run again. (Instead, skip to the step(s) after this section named “Gather data if needed”.)

(Updated process: instead of running the next command, go through the sub-process described by the section titled Getting “Key Header Info” For Processing, as going so helps determine whether a step is needed, which could save time compared to unnecessarily performing a rather time-consuming step. Once this guide's maintainer carefully ensures the following command does exactly match some information from that other section, it may be getting removed from here...)

cat $( ls -1t /tmp/eximbpid.* | head -1 ) | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}"; exim -Mvh "${arg}" ; done ' | tee $( mktemp -p /tmp eximkeyheaders.${MSGPATID}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"

Run these:

cat $( ls -1t /tmp/eximkeyheaders.${DISABLED/WAS/MSGPATID}-* | head -1 ) | nl | grep --color=always -Ei "\^\$"\|"====="\|" {2}-host_address "\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: "\|" X-Spam-Status: Yes, score="\|" X-Spam-Flag: " | tee $( mktemp -p /tmp eximrep.${MSGPATID}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
less -R $( ls -1t /tmp/eximrep.${MSGPATID}-* | head -1 )
Take Action

Take action on the sending E-Mail address

Suspend...

If it is clear that an E-Mail address has been sending lots of spam, then suspend that E-Mail address's ability to send messages.

  • There may be mutiple ways to do this:
    • This can be done within Exim.
      • For users of the poular software suite named “cPanel”, cPanel forum post: effects of “Suspend Sending Outgoing Email” documents “a custom Exim ACL is configured to block outgoing emails” by checking “the /etc/outgoing_mail_suspended_users file”.
        • (At the time of thsi writing, the author of this guide didn't yet determine whether that filename is commonly used on systems that aren't using the software suite named cPanel.)
        • cPanel users can use the cPanel administrative interface (not just WHM, but the interface that is often shared with customers) to control suspension. If you think a customer might unsuspend before changing the password, you might be better off changing the password. Sufficiently-new versions of cPanel can suspend just the outgoing E-Mail, but older versions might just need to fully suspend an E-Mail address which could also affect the ability to receive mail at that address.
      • For those who don't use cPanel, a guide is currently not here. cPanel forum post: effects of “Suspend Sending Outgoing Email” explained a bit how the feature works. It seems there is a file which contains Exim access control list (based on the file's path: /usr/local/cpanel/etc/exim/acls/ACL_RECIPIENT_BLOCK/default_recipient). The file contained a line that said, “deny”, and within that section, a line that said, “  condition = ${perl{check_outgoing_mail_suspended}}”.
        • Beyond this commentary, this guide does not currently provide (further) details (beyond what was just mentioned) about how to suspend just outgoing E-Mail for a specific address. Remember, if you aren't finding a way, changing the account's password may be effective.
    • Another way may be to change the user's password. (Doing so may do more than block the E-Mail address from sending E-Mail. Probably most systems are set up in a way that this would also prevent the address's intended user and the attackers from having access to other things like reading E-Mail. This might also affect other things, like the ability for the user and possibly the attackers to log into a computer system. Naturally, details will vary based on how computer(s0 are set up.)
  • This guide recommends doing this first, so that the account doesn't send more messages while trying to do anything else. (e.g., before trying to spend time clearing out a pre-existing backlog of queued messages)
Prevent...

Prevent existing spam from being sent. Some options include:

Removing Messages
  • This may be a ensible approach, especially if you have already generated a report showing subject lines. However, if you want to do a detailed analysis, e.g., possibly studying any executable code that might have been getting distributed, then you might want to preserve the messages for further handling/analysis. (Beyond presereving the messages, simply by not deleting them, and freezing the messages which is about to be discussed, this guide doen't go into much detail about handling and doesn't go into details on how to analyze such messages further.)
  • Here are two ways to do that:
Bouncing/cancel the messages
when to bounce/cancel

Note: This is still thoeretical...

Some spam-style attacks involve essentially tricking a computer into sending NDR/DSN (“non-delivery report”/“deliver status notification”) messages as a way to automatically get a computer to amplify bandwidth usage (at the notable expense of the E-Mail server doing the amplification). The NDR/DSN messages may include enough details to effectively send a (malicious) message to an intended recipient. To prevent getting involved in such Internet-based attacks, this guide recommends not bouncing on remote systems.

However, local senders may be suitable. (You can run hostname to get a name used by the current system.)

how to bounce local

The process recommended here is to only do this if the sender's E-Mail address is local. (This command does not check whether the sender's E-Mail address appears to be a local domainm, so such checks should just be done manually until another method is determined.)

cat $( ls -1t /tmp/eximgrp.${MSGPATID}-* | head -1 ) | xargs -L 1 exim -Mg | nl | ${SHELL} -c "time -p cat"
Deleting messages
when to delete

Note: This is still thoeretical...

Some spam-style attacks involve essentially tricking a computer into sending NDR/DSN (“non-delivery report”/“deliver status notification”) messages as a way to automatically get a computer to amplify bandwidth usage (at the notable expense of the E-Mail server doing the amplification). The NDR/DSN messages may include enough details to effectively send a (malicious) message to an intended recipient. To prevent getting involved in such Internet-based attacks, this guide recommends not bouncing on remote systems.

However, local senders may be suitable. (You can run hostname to get a name used by the current system.)

how to delete messages from the user

The process recommended here is to only do this if the sender's E-Mail address is local. (This command does not check whether the sender's E-Mail address appears to be a local domainm, so such checks should just be done manually until another method is determined.)

This change applies to messages

cat $( ls -1t /tmp/eximgrp.${MSGPATID}-* | head -1 ) | xargs -L 1 exim -Mrm | nl | ${SHELL} -c "time -p cat"
Freezing Messages
(More details forthcoming...)
  • Then, what do you want to do next? Handling the frozen messages might be a lower priority than some other response efforts, like getting off of public blacklists/etc. Having a very large number of frozen messages (around 176,000) seemed to lead to some slowness for Exim, but if that isn't an issue than additional response may be a bit of a lower priority. Other options could be:
    • review with the user of that address (determine if user wants to analyze messages, or if they could just be deleted),
    • or perhaps get them out of Exim's queue (maybe by routing messages, or maybe by shutting down Exim to move the message files).
      • (This guide does not currently elaborate on just how to do that.) Furthermore, Exim experts may indicate that this would be less preferable. This would likely induce downtime, and another approach could be to use Exim to route messages to somewhere othar than the active queue. However, this guide also does not provide details now about how to implement such an approach.
  • ideas...
    • exiqgrep -h shows help.
    • exiqgrep -i -z can be used on just frozen.
    • exiqgrep -i -x can be used on just thawed (meaning messages that are not currently frozen).
    • Could check how many of the messages are detected with “X-Spam-Status” / “X-Spam-Flag” or Subject of “***SPAM***”. (These ideas are discusssed more elsewhere...)
cat $( ls -1t /tmp/eximgrp.${MSGPATID}-* | head -1 ) | xargs -L 1 exim -Mf | nl | ${SHELL} -c "time -p cat"
Other...
  • Make sure the sending E-Mail address is noted. (This may be referred to later during some later follow-up step(s) that this guide recommends.)
  • At this point, a technician might need to make a decision on how to proceed next. In some cases, a technician might want to reach out to the person who the E-Mail address is assigned to, so that person can be informed of requirements and/or recommendations like changing the user's password and/or running anti-malware scans. It might also be prudent to just change the user's password, so that if the account somehow lost the freeze/suspension, that people with the existing compromised password wouldn't be able to start sending more spam.
(Repeat, as necessary...)
  • If there are more E-Mail addresses with a large number of E-Mail addresses in the queue, other than the local Mailer-Daemon account, then follow the above steps for the other E-Mail address(es).

    • If you need to review the list of high-sending messages again, you can re-build some of the data that was made before, or just try continuing off of some older data (if the data doesn't seem too old to be sufficiently accurate for making more progress). To use old data, the following command (which was listed earlier) may be helpful:

      less -R $( ls -1t /tmp/eximgrp.highsendat-* | head -1 )
    • Of course, if that data is too old, you could just re-create it by getting a more updated version of the data.
    • Follow above steps for handling E-Mail related to a single sender, starting with setting the variable and then choosing “Extract the data”. However, whenever seeing “patone”, use a different phrase (e.g. “pattwo”). (Actually, if you don't customize the variable, you might end up okay as sample commands may work with the most recent data. However, having multiple addresses using the same pattern could make things difficult to trace if any need/desire to do so appears.)

Here-io

[#eximqhmd]: Handle the Mailer-Daemon

The Mailer-Daemon may require some slightly different handling, primarily because Exim likes to refer to the E-Mail address as NULL (a.k.a. empty, e.g., “<>” instead of “Mail Delivery System <Mailer-Daemon@localhost>” or a more proper reference to the local server's host name).

Here is a reminder of how many message that likely involved (at an earlier point of time)...

See how many E-Mail messages are in the queue to be sent from the server's local Mailer-Daemon E-Mail address.

cat $( ls -1t /tmp/eximbp.* | head -1 ) | grep \<\> | cut -d \< -f 2 | cut -d \> -f 1 | grep -Ev "^ {7}.. " | uniq -c | ${SHELL} -c "time -p cat"
  • The parameter to ls is a one, not a lowercase L
  • The parameter seen in some grep commands is: a quotation mark, then a carrot, then ten (10) spaces before the next quotation mark.

Or, another way:

exiqgrep -if \<\> | wc -l | ${SHELL} -c "time -p cat"

(You might want to sanity check, by making sure that those two methods show the same number of messages.)

If you want to just get them out of the queue, one way is to bounce them. (Note that if you do this right away, you may lose the ability to see much more detail about what was in the queue.) To freeze them, you could use:

exiqgrep -if \<\> | xargs -L 1 exim -Mf | nl | ${SHELL} -c "time -p cat"

then, to bounce, instead of using -Mf, you could use: -Mg

(or, you could just bounce without freezing. For small numbers, where it is almost instantaneous, freezing first can just be a bit conforting, to make sure the number that gets reported frozen is the same as the number of messages that we were expecting to effect. If that number is surprising, you may want to re-evaluate before performing the bounce command.)

Checking for Returned Mail to Long Addresses

This search may be a bit time consuming and not producing results that get re-used in different ways. It looks for something very specific. On at least one server, this has been known to happen somewhat frequently with actual spam messags.

If your organization uses a “FirstName.LastName” pattern to E-Mail addresses, you might regularly have addresses exceeding 12 characters before the @. Like all advice in this guide, customize to suit individual needs and usefulness.

env MAXOKLEN=12 ${SHELL} -c "exiqgrep -i -f \<\> -r .\\{\${MAXOKLEN}\\}.*@.*" | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}"; exim -Mvh "${arg}" ; done ' | tee $( env MSGPATID=mailersoftw mktemp -p /tmp eximgrp.fmdtlong.${MSGPATID}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
  • This only shows messages where the recipient's name is twelve (12 or more characters long before the @ of the E-Mail address.

(Once the data is extracted with the prior command, these next two commands will likely go faster...)

cat $( ls -1t /tmp/eximgrp.fmdtlong.* | head -1 ) | grep -Ei --color=always "Subject: .{0,6}Mail delivery failed: returning message to sender" | wc -l | ${SHELL} -c "time -p cat"
cat $( ls -1t /tmp/eximgrp.fmdtlong.* | head -1 ) | grep --color=always -Ei "\^\$"\|"====="\|" {2}-host_address "\|"F From: "\|"T To: "\|" Subject: .{0,6}Mail delivery failed: returning message to sender"\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: "\|" X-Spam-Status: Yes, score="\|" X-Spam-Flag: " | less -R

The following may look similar to prior steps, except that instead of specifying the full E-Mail address (e.g. “Mail Delivery System <Mailer-Daemon@localhost>”), we will be using the information that Exim uses with this address, which is the shortcut “<>”.

Extract the data:

exiqgrep -i -f \<\> | tee $( mktemp -p /tmp eximgrp.mailer-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"

Some reports:

Basic report of system's Mailer-Daemon messages

Generate and view report for system's Mailer-Daemon address...

cat $( ls -1t /tmp/eximgrp.mailer-* | head -1 ) | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}"; exim -Mvh "${arg}" ; done ' | nl | tee $( mktemp -p /tmp eximgrp.mailacct-headers-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
cat $( ls -1t /tmp/eximgrp.mailacct-headers-* | head -1 ) | grep --color=always -Ei "\^\$"\|"====="\|" {2}-host_address "\|"F From: "\|"T To: "\|" Subject: Mail delivery failed: returning message to sender"\|" Subject: Undelivered Mail Returned to Sender"\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: "\|" X-Spam-Status: Yes, score="\|" X-Spam-Flag: " | tee $( mktemp -p /tmp eximrep.mailer-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
( cat $( ls -1t /tmp/eximrep.mailer-* | head -1 ) && echo System Mailer Data @ File: $( ls -1t /tmp/eximrep.mailer-* | head -1 ) ) | less -R
  • Astute people checking the command lines may notice searching for “Subject: Mail delivery failed: returning message to sender” (e.g., a message from Exim may show this) and “Subject: Undelivered Mail Returned to Sender” (e.g., a message from a remote system using Postfix probably looks more like this) and also “Subject: ”. Yes, it is true that searching for just “Subject: ” would end up matching on either of those two longer strings as well. However, searching for the longer strings will cause the entire longer strings to look more like a recognized match, thanks to the “--color=always”.
Reporting Common Patterns For Mailer-Daemon

These fit some common patterns.

  • To help vastly cut down on documentation that was largely redundant, and to make largely-redundant tasks more easily/obviously recognizable, these tasks have been split into two parts. (The first part describes a task and sets some variables, while the second part performs the steps that have a lot more redundant logic.)
Report: returned messages

You might find there's a fairly high number of messsages involving the system's Mailer-Daemon account. If you'd like to get a bit of a better idea of what those messages include, you may want to see just how many of them have the commonly-used subject, “Mail delivery failed: returning message to sender”. (That might still leave you with questions about why there are so many, but at least may give a feeling of having a bit of a better understanding of what's happening when you know a bit more about what so many of the messages are looking like.)

Step 1: Prep
SPATSRCH="Subject: .{0,6}Mail delivery failed: returning message to sender"
 SPATNAME=mailret
Step 2: Perform Base Pattern Commands
  • It is recommended, after performing the next step, that people plan to return to this location in the documentation in order to run some of the later commands (instead of skipping over them).
  • Follow the steps in the section called Base Pattern Commands.
Spam Reporting...

WARNING: at the time of this writing, the most recent testing of these commands were not producing any matches. (That might have actually been accurate, given what E-Mail was actually on the system.) Before proceeding, be advised that this section might need some improvement before it works well...

We have basically three approaches...

There are various ways to note some detected spam. Here are a couple of ways by looking at spam-related headers, followed by a way that looks at the subject line.

Using X-Spam-Status
Step 1: Prep
SPATSRCH=" X-Spam-Status: Yes, score="
 SPATNAME=xspmstat

Note: if that last command reported zero matches, then there may be little to no point in proceeding to look at the resulting data. (There is some thought that the results of later commands might find some false positive matches and produce some misleading results. That hasn't been confirmed yet, either way...)

Step 2: Perform Base Pattern Commands
  • It is recommended, after performing the next step, that people plan to return to this location in the documentation in order to run some of the later commands (instead of skipping over them).
  • Follow the steps in the section called Base Pattern Commands.
Using Subject

Recommended: This is probably reflecting some results of different anti-spam system, and so might not be completely redundant with the above.

Step 1: Prep
SPATSRCH="Subject:..\\\*\\\*\\\*SPAM\\\*\\\*\\\*."
 SPATNAME=xspmflag
Step 2: Perform Base Pattern Commands
  • It is recommended, after performing the next step, that people plan to return to this location in the documentation in order to run some of the later commands (instead of skipping over them).
  • Follow the steps in the section called Base Pattern Commands.
Using X-Spam-Flag

Recommended: Skip this. This appears to just be results redundant with “ X-Spam-Status: Yes, score=”. This is just included mostly as another easy example, and also in case of the possibility that some system is using X-Spam-Flag, yet at the same time is not also using X-Spam-Status.

Step 1: Prep
SPATSRCH="X-Spam-Flag: YES"
 SPATNAME=xspmflag
Step 2: Perform Base Pattern Commands
  • It is recommended, after performing the next step, that people plan to return to this location in the documentation in order to run some of the later commands (instead of skipping over them).
  • Follow the steps in the section called Base Pattern Commands.
[#exmspmpt]: Base Pattern Commands
  • These commands are not intended to be used until setting the SPATSRCH and SPATNAME variables!
    • If you haven't yet set those, check out the earlier instructinos which specify to set those.
    • These commands may output to files with names that may include the contents of some of the above variables.
    • (Sorry for the inconvenience of breaking up the commands like that, but it saved a lot of duplication.)
  • These examples use the /tmp/eximrep.mailer-* file, which expected to contains details abuot E-Mail messages from the Mailer-Daemon account.
    • (That file should have those types of messages because of some commands shown earlier in this guide).
    • If you were wanting to do similar things to messages from other users, you'll want to modify these examples to start looking somewhere other than the /tmp/eximrep.mailer-* file.
Extract Details of Focused-on Messages, & Count
SPATDATA=$( ls -1t /tmp/eximrep.mailer-* | head -1 )
cat ${SPATDATA} | grep -Ei "${SPATSRCH}" | awk "{\$1=\$2=\"\"; print substr(\$0,2)}" | tee $( mktemp -p /tmp eximgrp.${SPATNAME}-foundheader-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
cat $( ls -1t /tmp/eximgrp.${SPATNAME}-foundheader-* | head -1 ) | nl | wc -l
See details about reports of matching messages:
cat $( ls -1t /tmp/eximgrp.mailer-* | head -1 ) | xargs -d $'\n' sh -c 'for argtwo do echo "===== ${argtwo}"; exim -Mvh "${argtwo}" ; done ' | nl | grep --color=always -Ei "\^\$"\|"====="\|" {2}-host_address "\|" ${SPATSRCH}"\|"F From: Mail Delivery System <Mailer-Daemon@$(hostname)"\>\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: "\|" X-Spam-Status: Yes, score="\|" X-Spam-Flag: " | tee $( mktemp -p /tmp eximgrp-rep.${SPATNAME}-keyheaders-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
( cat $( ls -1t /tmp/eximgrp-rep.${SPATNAME}-keyheaders-* | head -1 ) && echo Report for -\"${SPATNAME}\"- File @: $( ls -1t /tmp/eximgrp-rep.${SPATNAME}-keyheaders-* | head -1 ) ) | less -R
Find Them (Again) & Handle Them
Freeze them:
echo ${SPATDATA}
 cat $( ls -1t /tmp/eximrep.mailer-* | head -1 ) | grep -i "=====" | cut -d K -f 3-14 | xargs -L 1 echo exim -Mf | nl

If you like what you see and what to proceed with running those commands as they are seen, you can simply take out the word “echo” that shows up just before the word “exim” (and re-run that last command after being modified as just described).

If, instead, you would like to...

  • ... thaw a message that was previously frozen: instead of “exim -Mf ”, use “exim -Mt ”
  • ... cancel/bounce: instead of “exim -Mf ”, use “exim -Mg ”
  • ... delete the message: instead of “exim -Mf ”, use “exim -Mrm ”
Reporting Detected Spam

WARNING: THIS SEEMS BROKEN at the time of this writing. (Check out the “Spam Reporting...” section, instead.) (This section may be removed if it is found to be redundant or lacking enough details to be useful in being abl to run a working command...)

There are various ways to note some detected spam. Here are a couple of ways by looking at spam-related headers, followed by a way that looks at the subject line.

Using X-Spam-Flag
==========
Extract Details of Reportedly Spammy Messages, & Count
cat $( ls -1t /tmp/eximrep.mailer-* | head -1 ) | grep -Ei "Subject: .{0,6}Mail delivery failed: returning message to sender" | cut -f 1 -d " " | tee $( mktemp -p /tmp eximgrp.mailret-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
cat $( ls -1t /tmp/eximgrp.mailret-* | head -1 ) | nl | wc -l
See details about reports of matching messages:
cat $( ls -1t /tmp/eximgrp.mailer-* | head -1 ) | xargs -d $'\n' sh -c 'for argtwo do echo "===== ${argtwo}"; exim -Mvh "${argtwo}" | grep -Ei "${SPATSRCH}" && echo -n FoundMatch-'${SPATNAME}'-:${argtwo}: ; echo exim -Mvh "${argtwo}" ; done ' | tee $( mktemp -p /tmp eximgrppatdat.${SPATNAME}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
cat $( ls -1t /tmp/eximgrppatdat.${SPATNAME}-* | head -1 ) | grep -i "^FoundMatch-${SPATNAME}-:" | cut -d : -f 2 | tee $( mktemp -p /tmp eximgrppatids.${SPATNAME}-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
cat $( ls -1t /tmp/eximgrppatids.${SPATNAME}-* | head -1 ) | xargs -d $'\n' sh -c 'for argtwo do echo "===== ${argtwo}"; exim -Mvh "${argtwo}" ; done ' | nl | grep --color=always -Ei "\^\$"\|"====="\|"${SPATSRCH}"\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" X-Spam-Status: Yes, score="\|" {2}-host_address "\|" Date: " | tee $( mktemp -p /tmp eximgrp-rep.mailret-XXXXXXXXXX ) >> /dev/null | ${SHELL} -c "time -p cat"
( cat $( ls -1t /tmp/eximgrp-rep.mailret-* | head -1 ) && echo Report -\"${SPATNAME}\"- File Data: $( ls -1t /tmp/eximgrp-rep.mailret-* | head -1 ) ) | less -R
Find Them (Again) & Handle Them
Freeze them:
cat $( ls -1t /tmp/eximgrppatids.${SPATNAME}-* | head -1 ) | xargs -L 1 exim -Mf | nl

If, instead, you would like to...

  • ... thaw a message that was previously frozen: instead of “exim -Mf ”, use “exim -Mt ”
  • ... cancel/bounce: instead of “exim -Mf ”, use “exim -Mg ”
  • ... delete the message: instead of “exim -Mf ”, use “exim -Mrm ”
=======================
[#eximqsrh]: Some additional searches

Note: When this documentation was updated and some/all of the commands in the next section were tested on a system that had thousands of messages on its queue, these results are not showing any matches. That could indicate that this documentation is broken, or maybe the spam received just didn't happen to match this.

Here are some additional searches. They may need to extract data, so they may be a bit more time consuming than many of the reports shown earlier.

Individual examples in this section might not be a standard universal practice that is widely recommended to try to perform. This may be more along the lines of... if you wanted to try to find this type of information, samples here may show how...

Messages with “bounce” in sender's name...
Count:
exiqgrep -i -f .*bounce.*@.* | nl | wc -l
Freeze:
exiqgrep -i -f .*bounce.*@.* | xargs -L 1 exim -Mf | nl
Remove:
exiqgrep -i -f .*bounce.*@.* | xargs -L 1 exim -Mrm | nl
Messages with long sender addresses in them...

Addresses with more than about a dozen and a half characters before the @ usually do not respresent addresses from E-Mail addresses that people use. These might be automatically-generated responses, or might be spammer's attempts to make up a unique address that won't match any blacklisted address.

These might be worthwhile, but viewing them before just deleting them is probably an especially good idea...

count:
exiqgrep -i -f \<.{20}.*@.*\> | nl | wc -l
disable (freeze)
exiqgrep -i -f \<.{20}.*@.*\> | xargs -L 1 exim -Mf | nl
Messages to specific TLD
  • First, here is a specific example that is also useful as a template if you want something a bit different...
  • From Mailer Daemon to specific TLD

    E-Mail from Mailer Daemon to specific recipient at specific TLD

    Count

    As an example here, show the number of messages to .guru and from Mailer-Daemon

    • A prior example here shows how to single out messages to a .casa domain. Note that .casa domains may be legitimate. (ICANN Wiki: .casa) This particular example was used just because this documentation was created while handling a series of incidents where there were a lot of spam messages being sent to .casa domains.
    exiqgrep -i -f \<\> -r @.+\\.guru | xargs -L 1 exim -Mvh | nl | grep --color=always -Ei "^$"\|" Subject: Mail delivery failed: returning message to sender" | wc -l
    • -f represents who the message is from. “-f \<\>” causes the shell to send “-f <>” to the program, where <> represents the “Mailer Daemon” account.
    • -r represents the recipient
    Disable (Freeze)

    As an example here, freeze the messages from Mailer Daemon, and to any domain ending with .guru (regardless of subject)

    exiqgrep -i -f \<\> -r @.+\\.guru | xargs -L 1 exim -Mf | nl
  • Then, if you want something a bit different:
    • If you had a different specific sender in mind, then just put that E-Mail address in between the inequality signs. e.g., instead of “ -f \<\> ”, use something more like: “ -f \<user@example.org\> ”
    • If you wanted to process all messages without concern of the E-Mail address that the message appears to be “From” (a.k.a. the sender), then just (entirely) remove the “-f \<\>” portion.
[#eximqfol]: Following Up
  • Once you believe that the offending E-Mail address(es) are not going to keep sending more E-Mail (e.g., maybe the ability to send has been suspended for just the E-Mail address(es) that caused problems earlier...)
  • Follow-up steps per offending E-Mail address
    • These are some steps recommended for each E-Mail address that was sending large amounts of spam...
    • Consider/handle where connections were from...
      1. Check: Identify where connections were from...
        • This can probably be done by looking at -hostname lines? (e.g., in the eximrep.* and/or the yet-more-detailed eximbp.* file(s) mentioned earlier in this guide.
          • (Such details are not included here...) (That might change in a future version of this guide. At the time this guide was being heavily worked on, there were some other details had seemed like a higher priority...)
        • If you are using Dovecot (which is a separate program from Exim, but is related to offering E-Mail servers and so it may often co-exist), that may also have a log file that might have some useful details. Some details of this were checked into before the creator of this guide noticed the usefulness of “-hostname” in the message (headers/logs) details. So, at least for now, some information on that is provided here. (Perhaps a future version of this guide may move such information, in which case such details will likely be hyperlinked to.) Even if the -hostname info above is found to work, maybe the approach of Dovecot's logs may be equally or more useful? (At very least, if such information is useful, it may be nice/worthwhile to know about.)

          • See if you can figure out where the connections are coming from. e.g.:, the following could be helpful for a system using Dovecot (which creates the /var/log/maillog file)...

            grep -i user@example.net /var/log/maillog | grep imap-login | grep -v rip=::1 | cut -d = -f 4 | cut -d , -f 1 | sort | uniq -c | sort -nr
            • Actually, the following might be usefully more restrictive:
              grep -i "Disconnected (auth failed,.*user=.*\>user@example.net\>" /var/log/maillog | grep imap-login | grep -v rip=::1 | cut -d = -f 4 | cut -d , -f 1 | sort | uniq -c | sort -nr
            • This is presuming a largely IPv4, and sorts the results based on octet. e.g.:
              grep -i user@example.net /var/log/maillog | grep imap-login | grep -v rip=::1 | cut -d = -f 4 | cut -d , -f 1 | sort | uniq -c | sort -nr | head -5
                   339 198.51.100.10
                    29 192.0.2.99
                    25 203.0.113.10
                    22 203.0.113.70
                    15 203.0.113.17

              Those were the top 5 IP addresses sorted by who was making the most connections.

            • Noticing several starting with similar numbers (e.g., 203.0.113.), this was adjusted a bit...

              grep -i user@example.net /var/log/maillog | grep imap-login | grep -v rip=::1 | cut -d = -f 4 | cut -d , -f 1 | cut -d . -f 1-3 | sort | uniq -c | sort -nr
                   447 203.0.113
                   339 198.51.100
                    29 192.0.2.99
                     2 #.#.#
                     2 #.#.#
                     1 #.#.#
                     1 #.#.#

              If we change the -f 1-3 to -f 1-4, we'd find that 339 came from a single address, and 447 came from various different addresses, all of which were IPv4 addresses starting with “203.0.113.”...

            • (A thought... Dovecot is included with computers using the popular software suite named “cPanel”. On such a system which also offers a feature named “WebMail”, does Dovecot get utilized by cPanel's WebMail, or is the WebMail feature of cPanel a different entry point that leads to access that doesn't get documented in Dovecot's logs?)
      2. (Determine if you want to block such IP addresses due to prior abusive behavior. Note that this guide does not elaborate here on how to do that. Since such actions are beyond the topic of this more-specific guide, this guide is not likely to provide significant levels of details here. Some hyperlinks/pointers might be provided by a future version of this guide.)
    • Following up with end user(s)...
      • An idea...
        • Some considerations before acting...
          • (Naturally, make sure you are authorized to make such a decision before proceeding with the idea here. Also, read the sub-point to help consider impacts before making a change, particularly if the old password isn't known by you and so you aren't able to reverse such an action.)
          • (as always, determine whether a change seems sensible before proceeding to make the change, especially if it is irreversible like not being able to restore an old password (due to not knowing it)...)
        • The idea itself:
          • You might want to just change the user's password, if that hasn't yet been done.
            • Presuming that it was compromised and abused, that now-problematic password should not be usable for sending out more E-Mail (ever again, since malicious bots might store that password indefinitely).
        • Some more considerations:
          • (If changing the E-Mail password was how the technician suspended the account's ability to send, then maybe this is already done...)
          • However, if the compromised password is still usable by the customer for incoming E-Mail, and outgoing E-Mail is blocked, then you might not want to further break the customer's access to incoming mail (by change the password)... Granted, in many cases where there was high spam, the user(s) of such an E-Mail address may struggle with handling incoming E-Mail anyway, because the incoming E-Mail may be swamped by “Non-delivery report” (“NDR”)/“delivery status notification” (“DSN”) messages. Still, some people might not like having functionality being blocked unnecessarily, so determine what is desired.
            • (A good excuse could be that the technician didn't want to leave the vulnerability open, in order to protect against any further possible intrusions on the privacy of any not-yet-compromised incoming E-Mail messages, including newer messages that didn't yet arrive before the password got changed.)
      • Inform the user(s) who may be impacted.
        • Note that they might not be able to easily notice an E-Mail from a technician if their incoming E-Mail has been, and might continue to be, swamped with incoming E-Mail messages which are “Non-delivery report” (“NDR”)/“delivery status notification” (“DSN”) messages. Other communication may be more effective.
        • Let them know what needs to happen and/or recommendations. For instance:
          • The customer should never use that password again. It should not be used for outgoing E-Mail, or any other purpose.
            • Some customers may be reluctant to make such a change. Clarify for them that a thief stole that password and has problematically abused it so far. Many people may have heard the advice, but are not familiar with another way to operate that seems sufficiently easy for them. Some people might benefit from a bit of training...
              • Many people re-use passwords extensively. (Studies have indicated that many people ignore advice.) It may be benefiical to introduce a user to a password manager (e.g. KeePass). The downside is that if the Password Manager breaks, they may lose quite a lot of stuff.
              • Writing down passwords is NOT a bad idea. What is a bad, which many people do, is write down passwords and then store such writings in an insecure way. However, if rarely-used passwords are stored with sufficient safety, then having them written may be acceptable. This can aid people in feeling comfortable with using different passwords for different things, which can reduce the pain of ceasing the use of a specific password.
              • https://www.goodfirms.co/blog/the-9-best-free-and-open-source-password-management-software discusses such software. Examples of passwod managers may include: KeePass, Bitwarden, yapet, pass (mentinoed by https://www.wired.com/story/best-password-managers/ and https://www.godo.dev/tutorials/text-based-password-manager/). Using browsers has often been not recommended, at least in part because such passwords have been known to be easily extractable/seen.
          • User is recommended to ensure a “full” vulnerability scan (e.g., using “anti-malware”/“anti-virus” software) is performed on every computer that may access the E-Mail account. Such measures should also be taken for any other device, like a phone, that may access the E-Mail account.
          • Presumably, giving the user a new password will be desirable.
          • Recommended: let the user know what to expect. After a spam incident, it may be common to receive additional “Non-delivery report” (“NDR”)/“delivery status notification” (“DSN”) messages, even after local clean-up is performed. That may continue for days, with the number of messages likely to reduce over time: most will be quick or within a few hours, but others may take 24 hours and others 48 hours and others 72 hours. This is caused by remote systems that may take a while before fully processing the message and responding, so the time frame is likely not something within the control or even influence of a technician overseeing the internal systems.
            • If this isn't explained, some users have been known to follow up with a statement that the problem (from the user's perspective, which may be getting flooded with “Non-delivery report” (“NDR”)/“delivery status notification” (“DSN”) messages) still isn't fixed.
        • Description of messages
          • End users may be able to get some idea of the subjects of the messages that were sent out, as such details are often mentinoed by resulting incoming “Non-delivery report” (“NDR”)/“delivery status notification” (“DSN”) messages.
        • Check desired handling of saved messages
          • If messages have't been deleted yet, determine if you want to check if the customer wants to analyze the messages. Some of the reports shown in the troubleshooting section might be sufficient (e.g., key headers), or perahps full messages would be preferable. (Obviously, try not to offer some details that are unavailable. For instance, if the messages have already been deleted, then sharing the messages may not be available. If some reports were generated earlier, those might still be salvageable as long as they weren't also deleted.)
          • (If the customer has no interest in saving the junk messages, then that may be one less reason to continue keeping them if they are still being kept.)
    • Document this
      • If you have incident documentation (e.g., a technicians' technical troubleshooting tracking ticket), it may be worthwhile to at least document what E-Mail address(es) have been involved. Some other details might also be nice, ike the number of messages that were sent and/or how many messages got stuck in the queue.
      • If you have documentation for an individual E-Mail address (or a customer's service account which provides E-Mail addresses for that customer), then such documentation may be nice/worthwhile to note. (If you also have detailed documentation elsewhere, such as incident documentation that contain details other than just the one E-Mail address, then a simply reference to that existing accessible documentation may be more prudent than duplicating all the details.)
  • Resend stalled messages
    • If there were messages that didn't get sent yet, but should be sent (and sooner may be preferable than later), then to ahead and get them re-sent.
    Sending messages
    • To resend a specific message
    • To send multiple messages:
      • The ElectricToolbox.com: “Flush the exim mail queue indicates one or more of the following may be helpful:
        runq

        This, or exim -q, may cause the entire queue to be re-processed. (That won't necessarily try re-sending something that is marked as being frozen. It just means that each item in the queue will have its status investigated again, so the message should be sent unless there is a detected reason not to send it, like if the message is frozen.)

        The runq command might be preferable as it may be a standard command name, usable on machines that use E-Mail servers other than Exim. However, it may also be less widely available. Machines using the software suite named “cPanel” have been known to not include some of the extra commands that seem to just be wrappers (like mailq, although runq did seem to exist.)

        exim -q

        This, or runq, may cause the entire queue to be re-processed. (That won't necessarily try re-sending something that is marked as being frozen. It just means that each item in the queue will have its status investigated again, so the message should be sent unless there is a detected reason not to send it, like if the message is frozen.)

        exim -qf

        Force queue processing of each message that isn't frozen.

        exim -qff

        Force queue processing of each message, even including frozen ones.

  • Good idea: make sure firewalls only allow outbound E-Mail traffic from authorized E-Mail servers. That way, if a workstation tries to send E-Mail without going through the authorized servers, that will be blocked. Since malware will often try that, blocking such communications is better than allowing to get on a blacklist.
  • Document what is desired (e.g., if there is a ticketing system that archives details, maybe include which E-Mail address(es) sent a lot of mail, sorted by message count (so as to also note how many messages), destination domains, etc. Maybe save reports (headers) of the messages. Maybe delete other details, like the information that was extracted with exiqgrep primarily just so that information could be processed to make the other reports; (a list of applicable Message IDs may be less relevant after those messages get deleted/delivered, so there may be no compelling point to document those...)
  • Clean up temporary files that this guide recommended making, but which are expected to not be of any use in the future.
    • See what exists:
      ls -l /tmp/exim{bp,grp}*
    • Determine if you want to make any sort of copy of those files, or leave the files there, or move them to another location (possibly to oblivion, a.k.a. deleting the files).
  • If alternate routing (e.g. cPanel Smart Host) was used, schedule a follow-up to verify removal from the blacklist and then, if desired, return from that alternate routing).

Some more information (may be merged into above steps here...)

  • Check for database corruption
    • https://www.thelinuxfaq.com/255-berkeley-db-error-panic-fatal-region-error-detected-in-exim says if /var/log/exim_mainlog shows: “Berkeley DB error: PANIC: fatal region error detected; run recovery”
    • then run:
      /etc/init.d/exim stop
      • Determine a unique filename. (e.g., using mktemp or...)
        • (For now, this sample command just uses a date, despite that practice possibly being something that is not recommended.)
      mv /var/spool/exim/db{>,bak-$( date +%Y%m%d%H%M%S)}

      Then restart Exim. (See: Restarting Exim.)

      https://myeasylinux.wordpress.com/2010/07/24/berkeley-db-error/ “BarkeleyDB error in exim logs”

Following is some older content. Some of that is expected to move into the above guide (somewhat/somehow, maybe changed a bit).

(Older Content)
Determine Biggest Sender(s)
grep @ | cut -d \< -f 2 | cut -d \> -f 1 | grep -vE "^ {10}" | sort | uniq -c | sort -nr
...
cat -i -f \<username@example.org\> | time -p tee $( mktemp -p /tmp eximgrp.patone-XXXXXXXXXX ) >> /dev/null
  • Generate a report showing some key information from headers. This can be helpful to determine whether the messages are spam, which may be helpful for the technician to make immediate decisions, and may be helpful if the sender of the messages wishes to question whether the removed messages were really spam.
    • Know Plan

    The basic goals are:

    • Pause (temporarily stop) outgoing mail
      • This might or might not be a good idea. For a very busy E-Mail server handling many E-Mail addresses, perhaps you don't want to shut off the flow for everybody while other steps are taken. However, if you can do this without problem, any reduction in outgoing spam is generally viewed as a good thing, and may help reduce the problems of reputation for your E-Mail server and/or it's IP addresses.
      • This guide currently does not go into detail on how to do this.
    • Determine what E-Mail accounts (addresses) are being used to send unwanted E-Mail
    • Stop that address from being able to send E-Mail
    • Clean up any further queued messages.
    • Re-enable outgoing E-Mail, which now should not send spam.
      • You may also wish to check if the system is on a notable public E-Mail Blacklist. Or, you may wish to delay that. More details about doing that are mentioned below.
    • Try to determine why that E-Mail address was sending spam.
      • If you are working for a mail provider (e.g., at an ISP), maybe you would best be off to outsource that task to whatever entity (organization/person) provides IT support for the customer.
    • Make sure the problem is not prone to happen again.
      • If a system was infected, make sure there is anti-malware protection.
      • Check history. If the same person/customer is repeatedly causing problems, make sure to know that this is not acceptable. Because the reputation
      • It is good to have a firewall which only allows outgoing E-Mail from authorized addresses, like a local server. Then, if a workstation gets infected and tries to send out messages, it is good to have internal IT Support be notified about the issue, but more to-the-point, the outgoing spam shouldn't reach the Internet.
    • See if the outgoing E-Mail server's IP addresses are on a notable E-Mail Blacklist.
      • e.g., check MX Toolbox (Blacklist section) and anti-abuse.org
      • Note: SORBS is a well-known blacklist which may include information about past events, so that isn't necessarily a strike. (At least, with some SORBS lists.)
      • Any blacklist that people should need to be concerned with should allow automated removal, or removal-by-request, as long as the problem wasn't too severe (e.g., there haven't been too many incidents).
        • This should be able to be done without submitting payment, nor providing contact details.
        • It might require submitting some information, like a statement of what happened, which may be used to evaluate future infractions. (Included in that “what happened” part is a statement about what steps were taken to prevent this from recurring.)
        • It might require a submission from the blacklisted IP address.)
        • It might take some time. A delay of over eight hours can happen; usually this gets resolved in 24 hours, although some longer time frames may be imposed, particularly if there is history with the E-Mail address, domain, and/or related IP addresses.
    • Possibly needed: See if the affected E-Mail addresses can send E-Mail just fine now.
    Check Server Overview
      • If you can identify an E-Mail address which is sending:
        • Put a hold on the ability for that E-Mail address to send.
        • (Determine if you want to block such IP addresses due to prior abusive behavior. Note that this guide does not elaborate here on how to do that...)
      • Possibly desired:
        • Make a reference as to some fo the details of the mail. e.g.:
          • Make sure spcified output file doesn't exist
            • The samples shown below use /tmp/spamlook.txt but that can be customized to a different filename if such a file already exists. For situations where an organization is using a tracking ticket, including the ticket number might help make output a bit more likely to be unique than these examples.
          • e.g.:
            exiqgrep -i -f \<username@example.org\> | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}"; exim -Mvh "${arg}" ; done ' | nl | grep --color=always -Ei "\^\$"\|"====="\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: "\|" X-Spam-Flag: " | tee -a /tmp/spamlook.txt | | less -R
          • Count how may messages were found:
            • grep -i ===== /tmp/spamlook.txt | time -p wc -l
            • Get messages out of the queue.
              • To freeze messages from a specific sender:
                exiqgrep -i -f \<username@example.org\> | xargs -L 1 exim -Mf | time -p nl
              • Once frozen, you may wish to determine whether to have them:
                • Remain frozen in the main quque
                • moved out to another folder, for review/safekeeping https://lists.exim.org/lurker/message/20130921.080247.eb5aabbb.en.html
                  • maybe by routing message to another location?
                  • Maybe by shutting down exima nd moving file(s) out of the exim-related folder(s)?
                • delivered (while frozen)
                • thawed, then delivered
                • Or deleted:
                  • e.g., to just delete the ones from a specific sender:
                    exiqgrep -i -f \<username@example.org\> | xargs -L 1 exim -Mrm | time -p nl
      • See how many are frozen
        • exim -bpr | grep frozen | time -p wc -l
          • Typically want low-- zero or near zero-- maybe a dozen at the most.
      • Show a summary
        • exim -bp | exiqsumm | sort -bgr | less
          • This summarizes messages, and can be a good way to identify some major trends like destination domains (If you want to see how well things have cleared up after some prior action, exim -bpc may be a better indicator for a simple count of what remains.)
      • Another summary:
        • Seeing a list of From/To: exim -bp | less
          • The following is a brief overview of what the information of the lines is believed to mean...
            ##h #.#s MsgId1-MsgID2-xx <?sender@example.com>
            ?recipient-of-frozen-mail@example.net
            D ?recipient-of-recently-delivered-mail@example.net
            • The first bit of time is how long ago the initial attempt was to deliver the message.
            • The next section indicates the size.
            • The third section (shown as “MsgId1-MsgID2-xx” in the above example) represent a message ID
              • The part between the < and the > represent who the E-Mail address is from.
              • That part between the < and the > might be blank (so it just looks like “<>”). Such a blank E-Mail address is actually short-hand for Mailer-Daemon@ (the local computer).
        • This may show both delivered messages and not-delivered messages. (If a single E-Mail message is sent to multiple recipients, this may show the messages which are delivered copies, as well as showing the messages that are undelivered copies.)
          • To not show the delivered mail, here is a variation: exim -bp | grep -vE "^ {8}D " | less
      • Reviewing queued outgoing messages (enlightening!)
        • This doesn't necessarily help to provide a quick overview, but it does let you see some details of the messages in the queue. If you don't know some of the details about the messaes and would like to quickly know what some of the messages are like, this shows you some key details about individual messages, and then lets you quickly see similar details about other messages.
        • This guide doesn't have any specific steps to do with the information, other than review what is seen to help determine whether it looks like the messages are probably spammy.
        exiqgrep -i | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}"; exim -Mvh "${arg}" ; done ' | nl | grep --color=always -Ei "\^\$"\|"====="\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" X-Spam-Status: Yes, score="\|" {2}-host_address "\|" Date: " | less -R
        • Naturally, don't be afraid to adjust. If you want fewer details, you can try to take out the relevant section fo the command line, or just append a “ | grep -v unwanted-header ” just before the “ | less ”. (Naturally, you would customize the “unwanted-header” portion mentioned there.)
        • exiqgrep -i | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}"; exim -Mvh "${arg}" ; done ' | nl | grep --color=always -Ei "\^\$"\|"====="\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|"spam"\|" Date: " | less -R
    Checking For Specific Message Patterns

    Experience has shown that sometimes a large number of messages can fit certain patterns. By identifying if that is the case, and then removing messages that fit certain patterns, you may be able to clear up much of the backlog quickly.

    In some cases, a specific pattern listed below may have many messages... dozens... hundreds... thousands...

    In other cases, some of the specific patterns shown here might not have many outbound messages queued up. Maybe even zero. It can still be good to know about these patterns, in case a backlog queue gets large in the future, because sometimes these patterns do successfully match a notable amount of messages.

    Mailer daemon: returning

    Messages from Mailer-Daemon with subject “Mail delivery failed: returning message to sender”

    count:

    See how many:

    exiqgrep -i -f \<\> | xargs -d $'\n' sh -c 'for arg do echo -n "${arg} "; exim -Mvh "${arg}" | grep -i "Subject:" ; done' | grep -i "Subject: Mail delivery failed: returning message to sender" | cut -f 1 -d " " | nl | wc -l
    More Details:

    See details about them:

    exiqgrep -i -f \<\> | xargs -d $'\n' sh -c 'for arg do echo -n "${arg} "; exim -Mvh "${arg}" | grep -i "Subject: " ; done ' | grep -i "Subject: Mail delivery failed: returning message to sender" | cut -f 1 -d " " | xargs -d $'\n' sh -c 'for argtwo do echo "===== ${argtwo}"; exim -Mvh "${argtwo}" ; done ' | nl | grep --color=always -Ei "\^\$"\|"====="\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: " | less -R
    Handling Identified Spam

    Handling messages already identified as spam:...

    There are multiple categories listed here. (Check each of them.) One is X-Spam-Flag, and the other is the ***SPAM*** text in a subject. Note that these may only find messages from the system daemon; specifying a different E-Mail address after -f (instead of the NULL address which refers to the system daemon) may produce different results, and just leaving off the -f <> entirely may show more results (due to not restricting output to a certain username).

    Part 1: X-Spam-Flag: YES
    count:

    See how many queued messages were already identified as spam

    • exiqgrep -i -f \<\> | xargs -d $'\n' sh -c 'for arg do echo -n "${arg} "; exim -Mvh "${arg}" | grep -i X-Spam-Flag ; done' | grep -i "X-Spam-Flag: YES" | wc -l
    More Details:

    Get Details about messages categorized as spam:

    exiqgrep -i -f \<\> | xargs -d $'\n' sh -c 'for arg do echo -n "${arg} "; exim -Mvh "${arg}" | grep -i X-Spam-Flag ; done' | grep -i "X-Spam-Flag: YES" | cut -f 1 -d " " | xargs -d $'\n' sh -c 'for arg do echo "===== ${arg}" ; exim -Mvh "${arg}" ; done ' | nl | grep --color=always -Ei "\^\$"\|"====="\|" X-Spam-Flag: YES"\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: " | less -R
    • This command is a bit redundant, searching for both “Spam” and “X-Spam-Flag”. That was done for easy customization/modifiability. The results of the command shown above can be prety verbose, and may be viewed as overly-verbose. Results can be trimmed a bit by removing (or even just inserting characters, to misspell) the ord “Spam” (located between the “ Subject: ” and the “less -R”).
    Terminate

    Removing those messages already identified as spam...

    Part 2: Known Spam-Identifying Text in Subject
    Count

    See how many queued messages were already identified as spam

    • Recommended command: exiqgrep -i | xargs -d $'\n' sh -c 'for arg do echo -n "${arg} "; exim -Mvh "${arg}" | grep -i "Subject:..\\\*\\\*\\\*SPAM\\\*\\\*\\\*." ; done' | cut -f 1 -d " " | nl | wc -l
      • Another variation (might work as well) exiqgrep -i | xargs -d $'\n' sh -c 'for arg do echo -n "${arg} "; exim -Mvh "${arg}" | grep -i "Subject:..\*\*\*SPAM\*\*\*." ; done' | cut -f 1 -d " " | nl | wc -l
    More Details

    Get details:

    exiqgrep -i | xargs -d $'\n' sh -c 'for arg do echo -n "${arg} "; exim -Mvh "${arg}" | grep -i "Subject:..\\\*\\\*\\\*SPAM\\\*\\\*\\\*." ; done' | cut -f 1 -d " " | xargs -L 1 exim -Mvh | nl | grep --color=always -Ei "\^\$"\|"====="\|" X-Spam-Flag: YES"\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: " | less -R
    Terminate

    Delete:

    exiqgrep -i | xargs -d $'\n' sh -c 'for arg do echo -n "${arg} "; exim -Mvh "${arg}" | grep -i "Subject:..\\\*\\\*\\\*SPAM\\\*\\\*\\\*." ; done' | cut -f 1 -d " " | xargs -L 1 exim -Mrm | nl
    Some other patterns

    After possibly clearing some of the above categories, it may be useful to re-run some of the earlier commands, like “ exim -bpc ”, which is described above. In particular, “ exim -bp | exiqsumm | sort -bgr | less ” may show some domains that receive a high number of messages. If there are notably large numbers of E-Mail messages that go to some foreign domains, which do not regularly get used for legitimate purposes, that may indicate that those messages are spammy.

    Specific Site

    If there is a particular site in question, you can check about E-Mails with that site as a recipient. (Perhaps this mainly works on domains other than the domain that the E-Mail server itself is in, e.g. so it might not be as suitable for a provider's main domain)...

    exiqgrep -r ".*@example.com"
    • Finds E-Mail messages going to E-Mail addresses at example.com
    • Shows: time in queue, file size, header ID, , and on next line(s), recipient(s) (until blank line). Although, doesn't show subjects (like the prior lines)
    • To show more info about a message, e.g.:
      • cat /var/log/exim_mainlog | grep -i msg-header-ID
        • Customize the “msg-header-ID” portion
        • If not found, you may want to check out some other log files. e.g.: perhaps use:
          • for x in /var/log/exim_mainlog-20??????.gz ; do gzip -dc $x ; done | grep -i msg-header-ID
    Sort by TLD
    (moved)
    Messages to a TLD
    As an example here, remove all messages (regardless of who they are from), and to any domain ending with .casa (regardless of subject) More aggressive: Remove all messages to .casa regardless of who the messages were from: exiqgrep -i -r @.+\\.casa | xargs -L 1 exim -Mrm | nl
         or perhaps: exiqgrep -i -r @.\*casa | xargs -L 1 exim -Mrm | nl
    Messages from System Administrator
    More aggressive: Remove all messages from System Administrator regardless of what they are about:
    count:
    exiqgrep -i -f \<\> | nl | wc -l
    More details:

    a bit more details...

    exiqgrep -i -f \<\> | xargs -L 1 exim -Mvh | grep --color=always -Ei "\^\$"\|"====="\|"F From: "\|"T To: "\|" Subject: "\|" Message-Id: "\|" X-Failed-Recipients: "\|" X-Mailer: "\|" Date: " | less -R
    terminate
    exiqgrep -i -f \<\> | xargs -L 1 exim -Mrm | nl
    Get more aggresive: Remove messages from Mailer Daemon
    exiqgrep -i -f \<\> -r @.+\\.guru | xargs -L 1 exim -Mrm | nl
    Other patterns to look for
    ....
    Follow-up steps
    • common step:
      • (as always, determine whether this is sensible before making a change, especially if it is irreversible like not being able to restore an old password)
      • ensure user's password is changed. Then, if sending was disabled, re-enable.
    Exim Message-IDs
    indicate a Message ID is basically in the format of:
    • TTTTTT-PPPPPP-tt
      • where TTTTTT refer to a number of seconds since a time epoch,
      • PPPPPP refers to a (presumably) unique PID of a process that received a message,
      • and “tt” refers to an amount of time (smaller than one second).

    You may find outbound messages stored in 4 files under /var/spool/exim/ as follows:

    msglog/<MsgID>
    Logged information about delivery details
    input/<MsgID>-H
    Message header info
    input/<MsgID>-D
    Message body data
    <MsgID>-J (presumably in input/)
    transient journals (described at https://lists.exim.org/lurker/message/20130921.080247.eb5aabbb.en.html ) (These are likely to only exist temporarily, so it is very possible, and likely even more desirable, if this doesn't exist... if it exists, that indicates work to be done, and once the work is done, this won't exist anymore...)

    https://electrictoolbox.com/show-exim-mail-queue/ also mentions some paths... as does BradTheMad.org's “Exim Cheatsheet”

    some earlier documentation had msgbox/f/ listed as the path. If that wasn't a typo, maybe that was just an example of some variation.

    misc info

    some misc. resource(s) which might be mentioned above:

    https://a1websitepro.com/exim-cheat-sheet/ “The examples given often pay no mind to the legitimacy of any message, they simply delete everything, assuming that if a message is in the queue then it must be junk.”

    https://www.geekdecoder.com/command-for-cpanel-exim-that-shows-script-responsible-for-outbound-mail/ indicates “cwd=/” may indicate a WordPress exploit

    See also; exiwhat?? (Mostly that seems to say the server is listening, althuogh if it is very actively processing a message then that can also be noticed.)

    Some more scratch notes

    To send messages, e.g. perhaps after unfreezing them (so they will get delivered eventually, but if you'd rather not have them be stalled/delayed unnecessarily any further), you may wish to try: