Spam Reduction Techniques Using greylisting and SpamAssassin

Spam Reduction Techniques

Using greylisting and SpamAssassin

The problem

The vast majority of email today is Spam Some current statistics indicate over 90% of

email Spam This matches my experience

Botnets

Vast majority of Spam comes from Botnets compromised home PCs hundreds of thousands to millions, or even tens

of millions of machines in a heard Controlled by the owner of the heard via a

centralised command and control structure Typically don't have a “real” smtp server to

actually send the email

Spam Reduction with Greylisting and SpamAssassin

Currently > 99% effective (closer to 99.8%)In a recent week, only 11 out of 8,000 Spam messages made it through to the end user without being stopped or marked.

Spam statistics as of: 16/09/2007Total spam: 5459Total greylisted: 4457(90.8%)Total emails accepted (both spam and legitimate): 451 (9.2)%Total identified spam through to end users: 1002 (20.4%)Emails greylist_delayed: 58 (1.2%), marked as spam 57 (96.6%), NOT marked as spam 2 (3.4%)emails via backup mx: 991 (20.2%), marked as spam 944 (95.2%), NOT marked as spam 48 (4.8%)Effectiveness of Greylisting / SpamAssassin: 99.0%. 50 out of 4908 not marked as spam




Greylisting removes > 90% of incomming Spam

SpamAssassin catches > 90% of received spam

Total effectiveness > 99.5%

Greylisting

Relies on Spammers not using a “proper” mail server. They just fire-and-forget

Give a temporary failure to any “suspect” messages. Spammers will not retry, but a mail server will

Which messages to challenge

Look at (all of): From address To Address IP of sending machine

If not seen before: give temporary failure record this “tuple” + time

If seen before: check if it is now past a “start time” (time + time to

go live) time to live is typically a parameter passed to greylisting

server. many recommend 60 minutes I use 60 seconds

OK – let through record the time

Not OK reject again

Any subsequent communication is let straight through

Some delay first time someone new contacts you

Small chance of non delivery of some messages. non compliant mail servers ISPs with rotary pool of mail servers may get

continually greylisted email from web forms that doesn't go through a real

mail server

Potential issues

Risk minimisation

Can have various white lists add mail server details for all regular / potential

contacts to a white list these emails are coming from a real mail server, so we

don't need to use this test on them. grep you mail server logs to determine who does conatct

you. eg:egrep "client=.*mail.*|client=.*mx.*|client=.*smtp.*" /var/log/maillog*| awk '{print $7}' | awk -F = '{print $2}' | awk -F [ '{print $1}' | sort | uniq -u

can use regex in these whitelists

Examples of server whitelist

/^.*\.ebay\.com$//.*\.emailebay\.com$//^.*\.mx\.bigpond\.com$//^.*\.dell\.com\.au$//^.*\.mailguard\.com\.au$//^mailout.*\.pacific\.net\.au$//^mail-out.*\.netspace\.net\.au$//^mx.*\.phx\.paypal\.com$//^smtp.*\.bis\.ap\.blackberry\.com$//^.*\.server-mail\.com$//^vscan.*\.westnet\.com\.au$//^ihug-mail\.icp-qv1-irony?\.iinet\.net\.au$/

Implementations

Available for many popular mail servers including MS Exchange

SpamAssassin

Categorises email as either Spam or Ham (good stuff, not Spam), based on a number of tests

Each test may add to the overall score for this email

If the total score exceeds a (configurable) limit, it is marked as Spam

Highly configurable personal limits, tests, scoring etc

Tests Tests to find words that look like viagra etc Is the sender in a RBL Does the sender match the SPF record

v=spf1 a mx mx:westnet.com.au include:westnet.com.au ~all

Does the body look like spam The ratio of text to images Bayesian analysis of the content Many more tests see: http://spamassassin.apache.org/tests_3_2_x.html

for the full list

Spam / Ham folders

can also set up folders containing Spam and Ham (non Spam) for SpamAssassin to learn from.

As a large proportion of email is actually spam (if you are not using greylisting), doing this may not be a good idea, as eventually the Bayesian filter gets poisoned and everything ends up looking like spam.

Implementations

Available for many popular mail servers including MS Exchange Exchange implementations tend to be commercial

offerings

SMTP Conversation

Client does: Server normally responds with: Connects to the server

250 Pleased to meet you MAIL FROM:<Sender address> 250 OK

250 OK

DATA

Sends the actual email message . 250 OK, accepted for delivery

220 Helo there HELO client-hostname

RCPT TO:<Recipient address> (May be repeated)

354 Start mail input; end with <CRLF>.<CRLF> (Nothing, it's waiting for the . that ends the message)

Greet - Pause

When the sender connects, delay the greeting If the sender tries to continue the conversation,

before the appropriate response, the conversation is stopped by the smtp server.

A “proper” smtp server will handle this, a Spam bot may just have a sequential script and fail this test.

About 10% of Spam can be eliminated this way

Components (in my system) Postfix mta (postfix-2.3.3-2)

http://www.postfix.org postgrey greylisting server (v 1.30)

http://postgrey.schweikert.ch/ See also http://www.greylisting.org/

SpamAssassin (spamassassin-3.2.2-1.el5.rf)http://spamassasin.apache.org/

http://postgrey.schweikert.ch/

http://www.greylisting.org/

http://spamassasin.apache.org/

Documents

Spam Reduction Techniques Using greylisting and SpamAssassin