System Aspects of Spam Control Architecture and Operations Issues IBM Academy 6 Apr 2005 Dave Crocker Brandenburg InternetWorking [email protected] IBM

System Aspects of Spam System Aspects of Spam ControlControl

Architecture and Operations IssuesArchitecture and Operations Issues

System Aspects of Spam System Aspects of Spam ControlControl

Architecture and Operations IssuesArchitecture and Operations Issues

IBM Academy • 6 Apr 2005

Dave CrockerBrandenburg InternetWorking

[email protected]

IBM Academy • 6 Apr 2005

Dave CrockerBrandenburg InternetWorking

[email protected]

D. Crocker IBM Academy / Spam Technical Issues22

Setting the ContextSetting the ContextSetting the ContextSetting the Context

© © 1975(!)1975(!)DatamationDatamation

This? Oh, this is the display for my electronic junk mail.


Approaching the TopicApproaching the TopicApproaching the TopicApproaching the Topic

Spam and email are complicated, global, human» We neeed a technical response to a social problem» Spammers are aggressive, bright, adaptive, well-organized» Nothing will “eliminate” spam! But we can control it.

Many points of control in the email architecture» We need a coherent framework for spam control» 1B users Simplistic solutions will be damaging

Assess Proposals carefully» Risk, cost, scaling, efficacy and durability» Local, transient effects spammers use different

techniques, versus» Global, long-term effects that truly reduce spam at its

core

Spam and email are complicated, global, human» We neeed a technical response to a social problem» Spammers are aggressive, bright, adaptive, well-organized» Nothing will “eliminate” spam! But we can control it.

Many points of control in the email architecture» We need a coherent framework for spam control» 1B users Simplistic solutions will be damaging

Assess Proposals carefully» Risk, cost, scaling, efficacy and durability» Local, transient effects spammers use different

techniques, versus» Global, long-term effects that truly reduce spam at its

core


Dangerous LogicDangerous LogicDangerous LogicDangerous Logic

“We have to do something now!”(Ignore any side-effects, or dismiss them as minor.)

“Maybe it’s not perfect…but at least we’re taking some action!”

“What have we got to lose?” “At least it reduces the problem…

for now.” “We must replace SMTP…

even though we don’t know what we want to do “We can do something in the interim…”

Even though nothing on the Internet is ever interim

“We have to do something now!”(Ignore any side-effects, or dismiss them as minor.)

“Maybe it’s not perfect…but at least we’re taking some action!”

“What have we got to lose?” “At least it reduces the problem…

for now.” “We must replace SMTP…

even though we don’t know what we want to do “We can do something in the interim…”

Even though nothing on the Internet is ever interim

“…“…but this is but this is urgent!!”urgent!!”


Examples of Solution Examples of Solution ChallengesChallengesExamples of Solution Examples of Solution ChallengesChallenges

Challenge/Response» Impose potentially large delay» Irritate legitimate senders & reduce

spontaneous communications

False positives » Lose sales opportunities and purchase orders

Reputation mechanisms» Can be gamed

Challenge/Response» Impose potentially large delay» Irritate legitimate senders & reduce

spontaneous communications

False positives » Lose sales opportunities and purchase orders

Reputation mechanisms» Can be gamed


Taking the Long ViewTaking the Long ViewTaking the Long ViewTaking the Long View

Imagine that time has passed…

Different types of spam» Accountable business will behave acceptably (mostly)» Rogue (criminal) spammers will be worse than today

Email» Will it still be easy to reach everyone?» Will it be cumbersome, with fragmented communities?» Will legitimate forms of communication be blocked?

Imagine that time has passed…

Different types of spam» Accountable business will behave acceptably (mostly)» Rogue (criminal) spammers will be worse than today

Email» Will it still be easy to reach everyone?» Will it be cumbersome, with fragmented communities?» Will legitimate forms of communication be blocked?


What is Spam?What is Spam?(and isn’t it impressive we need to ask this question!)(and isn’t it impressive we need to ask this question!)

What is Spam?What is Spam?(and isn’t it impressive we need to ask this question!)(and isn’t it impressive we need to ask this question!)

ChallengesChallenges

No clear community consensus on definition

» Strong on emotion» Weak on useful discussion

Minor, transient technical differences from good mail (!)

» Internet mechanisms are expensive to implement

» We must ensure they will quickly be effective for extended time

» Danger of arms race

ChallengesChallenges

No clear community consensus on definition

» Strong on emotion» Weak on useful discussion

Minor, transient technical differences from good mail (!)

» Internet mechanisms are expensive to implement

» We must ensure they will quickly be effective for extended time

» Danger of arms race

Sample DefinitionsSample Definitions

1. Whatever recipient decides

» This means we cannot provide institutional enforcement

2. Unsolicited Commercial» Religious, political, and

“crazies” are just as problematic

3. Unsolicited Bulk» Focus on consent/permission» Focus on aggregate traffic

Sample DefinitionsSample Definitions

1. Whatever recipient decides

» This means we cannot provide institutional enforcement

2. Unsolicited Commercial» Religious, political, and

“crazies” are just as problematic

3. Unsolicited Bulk» Focus on consent/permission» Focus on aggregate traffic


Universal Spam Solution Universal Spam Solution RebuttalRebuttalUniversal Spam Solution Universal Spam Solution RebuttalRebuttal

Checkbox form letter for responding to spam solutions proposals.

See:

<http://craphound.com/spamsolutions.txt>

Your post advocates a

( ) technical ( ) legislative ( ) market-based ( ) vigilante( ) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam.

Your idea will not work. Here is why it won't work.

(One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)…

Checkbox form letter for responding to spam solutions proposals.

See:

<http://craphound.com/spamsolutions.txt>

Your post advocates a

( ) technical ( ) legislative ( ) market-based ( ) vigilante( ) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam.

Your idea will not work. Here is why it won't work.

(One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)…


MHSMHS: Mail Handling Service

AU AU AU AU

AUAUAUAUAU AU AU AU

AU AU AU AU AU AU AU AU

Email ArchitectureEmail ArchitectureEmail ArchitectureEmail Architecture

MTAMTAMTAMTA MTAMTAMTAMTAMTAMTAMTAMTA MTAMTAMTAMTA

MAILMAIL : : MUAMUA = User= User MTAMTA = Transfer= TransferAGENTSAGENTS : : MSAMSA = Submission= SubmissionMDAMDA = Delivery= Delivery

MlistMlist = Mailing List= Mailing ListAUAU = Administrative Unit= Administrative Unit

oMUAoMUAoMUAoMUA rMUArMUArMUArMUAActorsActors ActorsActorsOriginatorOriginator

SenderSender DestDestRelayRelay

RecipientRecipientMediatorMediatorActorsActors

MDAMDAMDAMDAMSAMSAMSAMSA

BounceBounceBounceBounce

MTAMTAMTAMTA

MListMListMListMList


More Than One “Sender”More Than One “Sender”More Than One “Sender”More Than One “Sender”

MTAMTAMTAMTA MTAMTAMTAMTAMTAMTAMTAMTA MTAMTAMTAMTA

oMUAoMUAoMUAoMUA rMUArMUArMUArMUA

MDAMDAMDAMDAMSAMSAMSAMSA

BounceBounceBounceBounce

MTAMTAMTAMTA

MListMListMListMList

•MTA IP•MTA IP•HELO Domain•HELO Domain

•Provider Network IP•Provider Network IP

•Sender•Sender

•From•From

•Mail From

•Mail From

•Received•Received

•Sender•Sender


Email Points of ControlEmail Points of ControlEmail Points of ControlEmail Points of Control

ReactiveReactive

FilteringFiltering

ReactiveReactive

FilteringFilteringOriginatorUser Agent

Origin MailTransfer Agent External Mail

Transfer Agent

ReceiverUser Agent

Receive Mail Transfer Agent

External MailTransfer Agent

Proactive:Proactive:PricePriceAccountabilityAccountability

Reactive:Reactive:FilteringFilteringEnforcementEnforcement

Proactive:Proactive:PricePriceAccountabilityAccountability

Reactive:Reactive:FilteringFilteringEnforcementEnforcement


A Spamming “Architecture”A Spamming “Architecture”A Spamming “Architecture”A Spamming “Architecture”

SpammerSpammer

ZombieController ZombieZombieZombieZombieZombieZombie

ZombieZombieZombieZombieZombieZombie

ZombieZombieZombieZombieZombieZombie

Victim

InitialInitial WebWebPagePage

RevenueRevenue

WebWebPagePage


Wheel of Spam (Mis)FortuneWheel of Spam (Mis)FortuneWheel of Spam (Mis)FortuneWheel of Spam (Mis)Fortune

Control of spam» Cannot be “surgically” precise» Must balance the wheel» Needs range of partial

solutions» Different techniques for near-

term vs. long-term, except that near-term never is

Heuristics» Long lists complicated» Complicated Be careful!

Control of spam» Cannot be “surgically” precise» Must balance the wheel» Needs range of partial

solutions» Different techniques for near-

term vs. long-term, except that near-term never is

Heuristics» Long lists complicated» Complicated Be careful!

PoliticalPolitical

LegalLegal

SocialSocial

HumanHuman

AdministrationAdministration

TechnicalTechnical

ManagementManagement DeploymentDeployment

Many FacetsMany Facets


Types of ControlTypes of ControlTypes of ControlTypes of Control

Proactive Accountability

» Sender/Author» Sending host

Enforcement» Laws and contracts» Acceptable Use Policy» Scope of control?» Sufficiently objective

rules?» Avoids negative side-

effects

Proactive Accountability

» Sender/Author» Sending host

Enforcement» Laws and contracts» Acceptable Use Policy» Scope of control?» Sufficiently objective

rules?» Avoids negative side-

effects

Reactive (filtering)

Detection» Source or destination» Content» Aggregate traffic» Accreditation/Reputation

Action» Divert or delete» Label» Notification» Delay

Reactive (filtering)

Detection» Source or destination» Content» Aggregate traffic» Accreditation/Reputation

Action» Divert or delete» Label» Notification» Delay


Secondary ApproachesSecondary ApproachesSecondary ApproachesSecondary Approaches

Charging – Sender pays fee » Some vs. all senders» How much?» Who gets the money?

Enforcement – Laws and contracts» Scope of control – national boundaries?» Precise, objective, narrow?

Administration» Exchange filtering rules» Exchange incident (abuse) reports» Coordination among Abuse desks

Charging – Sender pays fee » Some vs. all senders» How much?» Who gets the money?

Enforcement – Laws and contracts» Scope of control – national boundaries?» Precise, objective, narrow?

Administration» Exchange filtering rules» Exchange incident (abuse) reports» Coordination among Abuse desks


Security FunctionsSecurity FunctionsMake someone accountableMake someone accountableSecurity FunctionsSecurity FunctionsMake someone accountableMake someone accountable

Term Function

IdentificationIdentification Who does this purport to be?

AuthenticatiAuthenticationon

Is it really them?

AuthorizatioAuthorizationn

What are they allowed to do?

AccreditationAccreditation What do I think of the agency giving them that permission?


Security ModelsSecurity ModelsSecurity ModelsSecurity Models

ObjectObject Channel (Path)Channel (Path)SecureSecure

MailMail

SecureSecure

MailSecureSecure

Mail

Mail

Mail MailMail

MTAMTA

MTAMTAMTAMTAMTAMTA MTAMTA

MTAMTAMTAMTASecureSecure

SecureSecureSecureSecure

MTAMTASecureSecure

MTAMTAMTAMTA

SecureSecure

MTAMTAMTAMTAMTAMTASecureSecure

MTAMTASecureSecure


SPF and Sender-ID:SPF and Sender-ID:Path RegistrationPath RegistrationSPF and Sender-ID:SPF and Sender-ID:Path RegistrationPath Registration

oMUAoMUA MSAMSA MTAMTA11

MTAMTA44 MDAMDA rMUArMUA

MTAMTA33

MTAMTA22

PeerMTA

PeerMTA

Assigns Sender & MailFrom

Did MSA authorize MTA1 to send this message?

Did MSA authorize MTA2

to send this message?

Did MSA authorize MTA3 to send this message?

1. Authority and Accreditation of MSA and MSA domain administrators

2. MSA must pre-register and trust each MTA in path


Beginning of CoherenceBeginning of CoherenceBeginning of CoherenceBeginning of Coherence

Validate content» DomainKeys, Identified

Internet Mail (IIM)» Transit signature of

msg

Validate operator» Client SMTP Validation

(CSV)» Operator validates MTA

Validate Bounce» BATV

Validate content» DomainKeys, Identified

Internet Mail (IIM)» Transit signature of

msg

Validate operator» Client SMTP Validation

(CSV)» Operator validates MTA

Validate Bounce» BATV

Reputation » CSA & DNA (CSV)» Still learning

Reporting» No candidates, yet

Enforcement» We are still learning

Reputation » CSA & DNA (CSV)» Still learning

Reporting» No candidates, yet

Enforcement» We are still learning


Certified Server Validation Certified Server Validation (CSV) (CSV) Assess Peer MTAAssess Peer MTA

Certified Server Validation Certified Server Validation (CSV) (CSV) Assess Peer MTAAssess Peer MTA

MUA MSA MTAMTA

MTAMTA MDAMDA MUAMUA

MTAMTA

MTAMTA

Peer MTA

• Does a domain's operator authorize this MTA to be sending email?

• Do independent accreditation services consider that domain's policies and practices sufficient for controlling email abuse?


Evaluating ProposalsEvaluating ProposalsEvaluating ProposalsEvaluating Proposals

Adoption» Effort to adopt proposal» Effort for ongoing use» Balance among

participants» Threshold to benefit

Operations impact on » Adopters of proposal» Others

Internet scaling – What if…» Use by everyone» Much bigger Internet

Robustness» How easily

circumvented

Adoption» Effort to adopt proposal» Effort for ongoing use» Balance among

participants» Threshold to benefit

Operations impact on » Adopters of proposal» Others

Internet scaling – What if…» Use by everyone» Much bigger Internet

Robustness» How easily

circumvented

System metrics» Cost» Efficiency» Reliability

Impact» Amount of Net affected » Amount of spam

affected

Test scenarios» Personal post/Reply » Mailing List» Inter-Enterprise

System metrics» Cost» Efficiency» Reliability

Impact» Amount of Net affected » Amount of spam

affected

Test scenarios» Personal post/Reply » Mailing List» Inter-Enterprise


How to Choose the FutureHow to Choose the FutureHow to Choose the FutureHow to Choose the Future

Look at each proposal» Who must adopt it? When?

» How much effort is need to administer it?

» How much does it change email?

Where to look for documents» ietf.org Internet-Drafts

» bbiw.net/current.html

Look at each proposal» Who must adopt it? When?

» How much effort is need to administer it?

» How much does it change email?

Where to look for documents» ietf.org Internet-Drafts

» bbiw.net/current.html

Documents

System Aspects of Spam Control Architecture and Operations Issues IBM Academy 6 Apr 2005 Dave Crocker Brandenburg InternetWorking [email protected] IBM