PerfectMail Tuning Guide - How To

Overivew

Because PerfectMail is mostly self-tuning, there are few tuning chores to distract administrators from their other duties. In fact, most PerfectMail appliances are run lights out (without administrator involvement) after their first few weeks of service.

PerfectMail's inherent accuracy is enhanced by its embedded reputation system. PerfectMail's reputation system helps ensure the highest overall accuracy (typically better than 99.9+% and zero false positives). PerfectMail auto-discovers protected e-mail users and peers as well as valid vs. malicious mail servers. It uses prior activity (from many perspectives) to help make the best overall decision.

This document discusses PerfectMail's scoring system. It provides insights into PerfectMail's scoring categories and how e-mail administrators can use PerfectMail reports to fine-tune PerfectMail’s domain settings so that messages are accurately and appropriately categorized while minimizing scoring uncertainty.

Message Scoring

Before attempting to tune PerfectMail, you need to understand how PerfectMail categorizes messages. PerfectMail uses three primary categories when scoring messages:

Accept    After being thoroughly scrutinized, the message was deemed wanted and is immediately forwarded to the intended recipient(s).

Reject     Messages that are rejected typically contain any of: unwanted content, obfuscated text, misleading or inaccurate e-mail header and/or envelope information, references to spam-friendly networks or other criteria that strongly indicates spam. As a result, PerfectMail refuses the message with an appropriate explanation to the sender. Reject messages are customizable so that in the unlikely chance the message was rejected in error, the sender can contact you by other means (phone).

Tag         PerfectMail tags messages that score above the Accept threshold but below the Reject threshold. Typically less than 1% of all messages are tagged.

Note: Messages containing viruses, unwanted file attachments, or known Phishing (fraudlent) messages are always rejected.

PerfectMail's default policy is to prepend the phrase [SPAM?] to the subject line of any Tag’d messages (customizable by the PerfectMail administrator). PerfectMail records the details of each message in its reputation system so that, as the sender’s reputation is established, PerfectMail will be less likely to Tag that senders messages.

Concerns can occasionally arise in your user community when a low frequency (or first-time) legitimate sender has receives a Tag score (and the [SPAM?] marker) on the subject line.

After a few weeks of services, administrators should take time to fine-tune PerfectMail so that the number of Tag’d messages is safely and accurately reduced. A few moments spent fine-tuning PerfectMail will result in a more pleasant experience for users (fewer Tags) and fewer support calls for administrators.

Default Scores

Administrators must assign default values for the Tag and Reject thresholds for each domain protected by PerfectMail. It is common practice to start with higher values, to ensure no false positives (legitimate mail rejected as unwanted), and then adjust values down over time. Higher initial values will allow some amount of unwanted e-mail (spam) to sneak in under the Tag and Reject scores. Determining and setting safe, long-term values for Tag and Reject can stop unwanted e-mail activity.

PerfectMail's reputation system will learn your users and their peers with a few days to a few weeks of service. Because PerfectMail strongly favors users and peers with an established reputation, it is safe to reduce Tag and Reject thresholds without the risk of introducing false-positive scores.

Optimal settings need to be determined empirically because each PerfectMail interacts with a unique set of users, mail peers and mail servers. To assist you with setting up your new appliance, XPMsoftware suggests the following settings based on our own experience with the product:

Deployment Type	Tag	Reject
Initial Deployment or for each new Domain	16	26
Retail ISP and non-business settings	14	24
Safe long term settings	12	22
More aggressive long term settings	11	18
Hiding the [SPAM?] marker	Same as Reject	22

The first thing to note is that scores have no meaning other than to indicate the magnitude of suspicious or undesirable activity discovered within a message. The overall range of scores that you might encounter is –50 or less (for messages between peers with well established history) to 50+ for messages from one-time senders of strongly objectionable content.

Initial Deployment

Unfortunately, some spam will score under 26 and may score under 16 so your users will still encounter unwanted messages. However PerfectMail is highly effective right out of the box, so the amount of unwanted messages should be dramatically reduced.

Retail ISP Settings

Safe long-term settings for ISPs and organizations that dealing with a mix of business and non-business traffic need to be set a little higher than for traditional business. If your user population is primary non-business (e.g.: a retail ISP), then you might want to try 14 & 24. For organizations that use e-mail as a business tool, slightly lower settings (perhaps 12 and 19) may be more effective.

Safe Long Term Settings

But, 12 and 22 are scores that appear to work in general and they may not be right for your system. Later in this document, we will explain how to review your system, use reports to help determine the optimal settings for Tag and Reject and how to apply your new settings to your domain(s).

Aggressive Long Term Settings

If your organization fits this description, you might want to consider setting your Tag and Reject thresholds to 11 and 17 respectively.

Note:      Do not reduce the Reject threshold below 11 without performing a thorough investigation to ensure that lower settings are safe for your organization. Our experience shows that some amount of e-mail, particularly from legitimate first-time senders may score up to 12. Use low Tag values only if you don’t mind first-time messages receiving the [SPAM?] marker on their subject line or you choose to hide the [SPAM?] marker.

Hiding the [SPAM?] Marker

Note:      PerfectMail provides an alternative (and preferred) method for achieving the same result. In PerfectMail’s web interface, select Server Config ® Misc. Items and uncheck Tag Subject Lines. This will prevent PerfectMail from inserting any indication that a message has exceeded the Tag threshold.

Reviewing E-Mail Scores

PerfectMail provides a real-time, interactive query and reporting facility that lets you examine activity by time, domain, user and peer. Of all of these reports, Query by Time and Query by Domain are the most useful in tuning PerfectMail.

Query By Time

To query by time, select Activity -> By Time. You should see the following selection screen:

E-mail Address will let you select only e-mail activity originating from or destined to a particular user. Leave this field blank to review sending and receiving activity across all users.

Warning: Be conservative when selecting starting and ending times. PerfectMail will fetch one record from its reputation system for each e-mail message, regardless of its Accept, Tag or Reject status for each message handled between the start and end times. On high volume mail systems this could result in the retrieval and display of many (tens of) thousands of records over a single day. Depending on the appliance you purchased, and the number of records selected, it may take PerfectMail a few seconds to a few minutes to fetch all of the requested records. Please be patient (especially if you selected hours or days worth of traffic).

It is best to start with a 10 or 15 minute interval. Simply adjust the start time hour and minute values back by the desired time and click Select.

PerfectMail’s reports may be sorted by all column headers (blue hyperlinks). By default, records are sorted by time (oldest to newest). Clicking any column header resorts the selected records by the appropriate field (in ascending order). Clicking the same column header a second time resorts the same field in descending order.

Since most servers receive much more legitimate e-mail traffic than Reject’d traffic, it makes sense to click the Score column header twice (to sort descending).

Warning: Rejected messages with exceptionally high scores often contain profane or adult references with possibly disturbing text. PerfectMail makes no attempt to hide this text. Viewer discretion is advised.

Note:      If you have selected a large number of records, it may take PerfectMail as much time to re-sort the data as it took to retrieve the records. Please be patient.

Use the browsers vertical scroll bar to scroll down through the report until you start to see records with a Tag status. At first, you will encounter messages that obviously should have been rejected – but were Tag’d and forwarded instead. Messages that should have been rejected will contain obviously unwanted subject lines, and may also have obviously invalid sender e-mail addresses (e.g.: fjksopkoksda@anydomain.com). In this example, no real person would reasonably select such an e-mail name.

Make a note of the highest score of the first e-mail message that should not obviously have been rejected.

Highest score of the first message that should not obviously have been rejected: _____

This score plus 1 or 2 is a good candidate for your new system-wide default Reject threshold. We will see how to apply this score to all domains a little later in this document.

Continue to scroll down through the Tag’d messages, ignoring any TmpFail’d messages you encounter.

Note:      TmpFail'd messages are messages where PerfectMail requested verification from the sending mail server. They are typically delayed for no more than a few minutes. The additional verification significantly improves PerfectMail’s accuracy on such messages.

Next, look for highest scoring message regardless of status that is obviously wanted. It is likely that you will have to look well into the Accept’d messages before you find a message whose subject line and sender appear to be completely legitimate. If in doubt, err on the side of caution and select the first plausible message that should be Accept’d.

Highest scoring message that is reasonably legitimate: _____

This score plus 1 or 2 is a good candidate for your new Tag setting.

Repeat the above exercise looking for good candidates for your Tag and Reject settings. See if you can’t arrive at a consensus across three to five different fifteen-minute intervals.

Enter your observations here:

Applying Your New Site-Wide Settings

1. Log into PerfectMail as admin or another account you defined
2. Click Domain Config
3. In the form, enter your Highest Score for both Tag and Reject into the form
4. Click Modify Domain

After a short delay, you should see your settings applied to all of the domain(s) defined on your system. These settings take immediate effect. You do not need to reboot PerfectMail or perform any other tasks to have your new settings applied to any new messages you receive.

If you have very few domains or, your are satisfied that all domains require approximately the same settings – then you are finished tuning PerfectMail. Otherwise, continue fine-tuning PerfectMail by conducting further setting analysis on a domain-by-domain basis (below).

Adjusting PerfectMail's Domain Settings

Some domains may receive a greater amount of higher scoring legitimate e-mail than others. To ensure that all users receive effective mail filtering you may need to conduct a domain-by-domain analysis of e-mail traffic to ensure that each domain is optimally set.

PerfectMail does not currently have a by domain view of mail traffic (it has a by domain/user/peer view and a by time view). So, to review traffic by domain, follow these steps:

1. Log onto PerfectMail as an admin user
2. Click Drill Down
3. Click a domain that you believe might be a good candidate for further tuning
4. Update the display so that output is in Text mode (Graphic/Text ® Update)
5. Click the Tag’d column header to sort by the maximum number of Tag’d messages
6. Click on the account name that accumulated the most Tag’d messages
7. Review the messages looking at the subject line, score and sender.

Repeat this process, filling in the form below for 5 or 6 users. Record the highest score for any Tag’d message that appears to be legitimate. Also enter the lowest score for any Tag’d message that is obviously Spam. Enter your observations here:

Use the data (above) to determine a safe Tag value (Highest Tag Legitimate + 1), and the Reject threshold (Lowest Tag Unwanted value + 3). Once you are satisfied that you’ve got the settings correct, update the domain with your custom settings. Complete these steps:

1. Click on Domain
2. Scroll down until you see the domain you just checked. Click that domain name.
3. Change the Tag and Reject values with the values derived above
4. Click Modify Domain

Warning:   Do not pick too low a Reject value or you may end up rejecting wanted messages (from first time senders). Such rejected messages cannot be retrieved.

Conclusion

PerfectMail provides data reporting and query tools that make it easy to determine the optimal settings for your appliance. These settings can be applied across all domains by simply updating a form.

But, as domains may experience vastly differing amounts of spam, you should consider doing a further analysis and tuning on domains if user complaints or the level of uncertainty indicate that such tuning is warranted.