02/07/26 1

Autonomix:Autonomic Defenses for Vulnerable Software

Crispin Cowan, Ph.D

WireX Communications, Inc

wirex.com

02/07/26 2

Talk Outline

• Progress: LSM

• Experimentation: Defcon

• Technology Transition

• Conclusions

02/07/26 3

LSM: Linux Security Modules

• Linux’s open source & broad popularity make it a great target for security research– SubDomain, DTE,

SELinux, etc.

• But one has to have a custom kernel to use these packages

• Solution: security modules for Linux– Standardized interface

in the Linux kernel for security modules

– Get Linus et al to adopt LSM

• Expected result:– Can load advanced

security into standard Linux kernels

02/07/26 4

LSM: Linux Security Module

• Unfortunately, none are standard to Linux– Maintained as kernel patches– To deploy them, must acquire a custom kernel

• Linus would like to support advanced security policy, but not willing to endorse one project.– Too political… “My security policy is better than yours.”– Linus is not a security expert, and doesn’t want to be– Linux is about choice anyway

• Solution: enrich Linux’s module interface to support security policy modules

02/07/26 5

LSM Design

• syscall interposition, i.e. wrappers at the syscall interface– not appropriate: leads to module bloat– already available by re-writing Linux syscall

table

• Instead, we mediate access to internal kernel objects“May subject X access object Y for operation Z?”

02/07/26 6

LSM - Architecture

User-level process

KernelLSMModule

Open syscall•Std. error checks•Std. Security checks•LSM hook:•Complete request

Policy engine•examine context•does request pass policy?•grant or deny

02/07/26 7

LSM - Architecture

User-level process

KernelLSMModule

Open syscall•Std. error checks•Std. Security checks•LSM hook:•Complete request

Policy engine•examine context•does request pass policy?•grant or deny

02/07/26 8

LSM - Architecture

User-level process

KernelLSMModule

Open syscall•Std. error checks•Std. Security checks•LSM hook:•Complete request

Policy engine•examine context•does request pass policy?•grant or deny

“ok with you?”

02/07/26 9

LSM - Architecture

User-level process

KernelLSMModule

Open syscall•Std. error checks•Std. Security checks•LSM hook:•Complete request

Policy engine•examine context•does request pass policy?•grant or deny

“ok with you?”

Yes or no

02/07/26 10

Hook Style

Restrictive: module may only reject a request about to be granted

Permissive: module may only permit a request about to be rejected

Authoritative: module may totally over-rule standard kernel logic

• We chose restrictive hooks only, except for capabilities– Simplifies LSM patch for maximum acceptability to Linux

community

02/07/26 11

Module Stacking

• Strong desire to compose modules• However, composition in general is intractable• Solution: stacking left to modules that want to

stack– Stackable module must export an LSM-like interface

“out the back”– Stackable module responsible for composing policy by

taking down-chain module’s results under advisement– Module-stacking module (MUX) in development

02/07/26 12

Hook Location

Structure ObjectTask_struct Task (process)Linux_binprm ProgramSuper_block FilesystemInode Pipe, File, or SocketFile Open FileSk_buff Network Buffer (Packet)Net_device Network DeviceKern_ipc_perm Semaphore, Shared

Memory Segment, orMessage Queue

Msg_msg Individual Message

02/07/26 13

LSM: Linux Security Module Progress Since February• Implemented & working

• Running various WireX servers + other LSM activists

• SELinux shipping exclusively LSM packages

• Heard all that last PI meeting ...

02/07/26 14

LSM Progress

• Paper presented at USENIX Security– Plus IBM has a paper on LSM hook placement

correctness

• Paper presented at Ottawa Linux Symposium– Plus three other papers with LSM content

• We were invited to the Linux Kernel Summit– Linus has accepted LSM;– Linux 2.5.27 has LSM in it

02/07/26 15

LSM ToDo’s

• Cut the LSM patch into bite-sized pieces– Easier for the Linux maintainers to digest

• Work with Al Viro to get the VFS patch we need– He’s working on it, but not quickly

• Address the network performance problem– Leverage network hooks out of LSM– It’s Netfilter’s fault

02/07/26 16

LSM Lesson:How to Get a Feature Into Linux• Linux allows you to do it your way, but to be in Linus’

kernel, you have to do it in a way acceptable to Linus– Do something that makes him happy

– Linus trusts his major subsystem maintainers, so work with them

– Keep someone interacting constructively with the LKML (Linux Kernel Mailing List), especially core developers

• Have to actually solve a problem– LSM is effectively a direct response to Linus wishing out

loud

02/07/26 17

LSM Lesson: Hard Choices

• To keep Linus happy, we had to make some tough choices– Security people largely would prefer authoritative hooks,

and many more of them– That would enable full POSIX audit logs

• But that also would have killed LSM’s chances of getting into Linux– Linus (and many others) do not like BSM– They would have regarded LSM as unnecessary bloat

02/07/26 18

LSM Lesson: Collaboration

• Before LSM:– Competing security projects all working independently

• With LSM:– Lots of collaboration among many projects– Working to provide a common infrastructure– Creating a common market for composable and competing

security features

• The trick:– Finding the right layer to abstract– Political engineering: make sure there is something in it for

everyone

02/07/26 19

LSM Collaboration Community

• 500 people on the mailing list

• Major contributions coming from:– 3 IBM sites (2 LTC sites +

T.J. Watson Lab)– SELinux (NAI/NSA)– SGI– Assorted open source

people

• Commercial LSM modules in the offing:– WireX:

• SubDomain• RaceGuard• CryptoMark

– HP Secure Linux– Ericsson Research

02/07/26 20

Experimentation ...

• Some real-world red teaming• Play an Immunix server in the Defcon

Capture the Flag (CtF) games• Almost no holds barred:

– No flooding– No physical attacks

• New gaming rig designed by the Ghettohackers

02/07/26 21

Basic Defcon CtF Rules

Player Nodes

02/07/26 22

Basic Defcon CtF Rules

Player Nodes

Score’botPolls player nodes,Looking for req. services

If all services found ...

02/07/26 23

Basic Defcon CtF Rules

Player Nodes

Score’botPolls player nodes,Looking for req. services

If all services found,Score one point for theFlag currently on thatnode

02/07/26 24

Basic Defcon CtF Rules

Player Nodes

Score’botPolls player nodes,Looking for req. services

If all services found,Score one point for theFlag currently on thatnode

… while each teamtries to replace others’ flags

02/07/26 25

No Flooding

• DoS attacks are not interesting• Explicit rule against flooding attacks

– Game masters will make you stop if you are caught at it

– Goal: ensure that all teams are actually able to play

• Penalties:– Kicked out for overt DoS attacks– Pay for bandwidth with a point penalty

02/07/26 26

Area View

02/07/26 27

Sporting Event

Teams named funky colors

Score obfuscated

• There was an official bookie :-)

• Score broadcast on hotel cable

Immunix was white,hence “Weiss Labs”

02/07/26 28

The Catch

• The required services are secret

• Only a few clues:– They supply us with a VMWare/Linux image

reference distribution that provides all required services

• It is also riddled with vulnerabilities

– The score’bot polls for the required services• But the score’bot stops its poll if it finds something it

doesn’t like

02/07/26 29

The Reference Distribution

• Red Hat 6.2, unpatched

• nmap: shows nearly everything open– finger, POP, IMAP, SMTP, SNMP, Webmin ...

• Apache running as root

• CGI’s for adduser and deleteuser– Anonymous can create a user login on your

node– As any user number, including zero

02/07/26 30

Example Services the Score’bot Wanted• Create a user• Send that user mail• Finger the user• POP in to fetch the mail• Delete the user• Note: no crypto protocols

– No proper authentication of the score’bot– Must heuristically distinguish score’bot from attacks

using behavior signatures

02/07/26 31

Interesting Challenge

• Not just survive severe attack, but also– Protect bad code– A lot of it– Vague functional specification– Rapid deployment

• Great new game infrastructure from Ghettohackers– Interesting challenge– Engaging scoreboard

02/07/26 32

Captain’s Meeting

• Explain the rules in detail

• Hand us the reference distribution

02/07/26 33

Setting Up

02/07/26 34

The Popular Strategy: Human Intrusion Detection

• Launch the reference Linux distribution

• Ad hoc patch as stuff happens

• Defend:– look for logins, I.e. non-score’bot behavior– kill them off ASAP– very labor-intensive

02/07/26 35

The Immunix Strategy: Protect Bad Code with Immunix Tools

• Port all plausible services to Immunix 7+ distribution– Use our own fingerd, httpd, etc., up-to-date and

compiled with StackGuard and FormatGuard– Run on an Immunix kernel with SubDomain and

RaceGuard– Wrap vulnerable services & CGI’s with SubDomain

profiles to limit access to least privilege necessary

• Launch only when we were reasonably confident that the Immunix machine was configured securely

02/07/26 36

Dealing with Logins: the SubDomain Shim

• Change adduser CGI to use a special default shell: /bin/fubush– /bin/fubush is just a hard link to /bin/bash– Restrict /bin/fubush to only the operations

needed by the score’bot

• Attackers can go ahead and create a login with uid 0 and it still won’t do them any good – They get a root shell, stuck in a tiny sandbox

02/07/26 37

Immunix Team

02/07/26 38

Immunix Team

Me

Chris Wright

SethArnold

Steve Beattie

• Plus 15 volunteers

02/07/26 39

From Our Corner

02/07/26 40

From Our Corner

JohnViega

Me

ChrisWright

SethArnold

SteveBeattie

02/07/26 41

Mental Stress

• This is a tough game to play– Head-to-head competition with a lot of very smart

people– Real-time– Continuous

• The intensity of qualifying exams– That go on for 22 hours in a 48 hour period– … set in the middle of a rave

• Hydrate or die :-)

02/07/26 42

Rave

• Loud music• Smoking• Gawkers• Social

engineering• Periodic

“news breaks”

02/07/26 43

Our Strategic Error

What We Did• For first 4 hours

– No server at all

– Porting services to Immunix ASAP, based largely on nmap and source inspection

• Next 4 hours– Launch Immunix server

– It’s secure, but is not making the score’bot happy

• Cost us massive points– Too focused on the science of

“can we defend Immunix?” and not enough on the game rules

What We Should Have Done• Launch reference system

immediately– Defend ad hoc like everyone

else

– Run network sniffer to determine what the score’bot wants

• Would have:– Put us over the top on points

– Learned what score’bot wants much faster

• We eventually did this

02/07/26 44

Immunix Server Not Up Yet

6th

place

02/07/26 45

Once Immunix Server Up …in the Score’bot’s Opinion :)

• Our score quickly rose

2nd

place

02/07/26 46

Once Immunix Server Up …in the Score’bot’s Opinion :)

Close2nd

place

02/07/26 47

Once Immunix Server Up …in the Score’bot’s Opinion :)

1st

place• Stayed

there most of Saturday

02/07/26 48

Late Saturday:New Service Requirement

• With 4 hours of play to go, the score’bot changed: now it wanted Webmin– Open source web-GUI for Linux administration– Competitor to WireX’s commercial server

appliance software– Rather famously vulnerable :)

• Took us 2 hours Sunday morning to make the score’bot happy again– Lost our lead

02/07/26 49

Some of Our Creative Attacks

Lock Out the Owner• Once we root the

machine, install a back door

• Also replace root’s login shell with /sbin/halt– Owner can’t log in to

their own machine– But we can

Spam’bot• Add user to their server• User sends spam mail to

all the other teams• Costs them penalty

points• Penalties are per

connection– Spam’bot sends 1-byte e-

mails

02/07/26 50

Final Score: 2nd Place

Team Score Points PenaltiesOrange 54.3764 64 9.6236White(Immunix)

51.1160 55 3.8840

Brown 48.2203 90 42.7797Green 40.1943 46 5.8057Yellow(Chaos Computer Club)

22.1865 43 20.8135

Red(Naval Postgraduate School)

6.1215 17 10.8785

Blue -22.7039 30 52.7039

Purple -24.5107 5 29.5107

02/07/26 51

Lesson: Symmetric Red Teaming Solves Rules Issues

• Everyone is both an attacker and defender

• Bad: everyone needs to learn how to attack

• Good:– Everyone should learn how attacks are done :-)– Rule fussing about how hard or easy it is for the

attacker apply to all parties -> less fussing

• Ghettohackers have designed a great game– Looking for technology transfer to Government

02/07/26 52

Lesson: Mandatory Access Control is Not Enough • telnetd was a required service• WireX never bothered to patch a vulnerability in telnetd for Immunix– Only idiots run telnetd :-)

• Someone hacked our telnetd– Didn’t get out of the SubDomain sandbox

– Did make our telnetd stop working

– Cost us a point that round

• General case: MAC protects your system, but not your individual services

02/07/26 53

Lesson: Resource Management is a Security Attribute

• SubDomain confined attacker logins to only run prescribed code– Including PERL

• Attacker launched a PERL fork bomb– Consumed all of real and virtual memory– While our machine is thrashing, the score’bot

passes us by– Costs us a point that round

02/07/26 54

Lesson: Redundancy Helps When You Are Vulnerable

• Penetration attacks take a long time to recover– Must clean up state, find & fix vulnerability

• DoS attacks take a long time to recover– If machine crashes, must fsck file system; can

take 10 minutes

• Hot spare can be on-line in seconds– Heterogeneous hot spare keeps attacker from

immediately deploying the same attack

02/07/26 55

Lesson: Redundancy is Resource-Constrained

• Must have humans on watch to clean up the compromised machine– The hot spare will not protect you for long

• Presumption that hot spare prevents attacker from attacking again assumes resource limit at the attacker’s end– If attacker has lots of exploits/resources, they

will hack your heterogeneous server just as quickly

02/07/26 56

Lesson: Immunix was Impenetrable, but not Incorruptible

• No one ever “flagged” the Immunix server– Others did plant enemy flags on our reference

server (as expected)

• But they did hit the Immunix server hard enough to compromise availability– Take out one required service, and the

score’bot doesn’t award a point– We missed first place by less than 4 points out

of approx. 55

02/07/26 57

Technology Transfer

LSM: Linux Security Module– Linus said “yes”

Immunix: licensed to Compaq/HP– Build a product family of security appliances– Firewall, AV Mail server, NIDS, etc.– Compaq hardware– Immunix OS, user interface– 3rd party security applications– Press release this week

02/07/26 58

Summary

• LSM:– Technical work stable– Political breakthrough– Technology transfer heavy lifting coming next; feed

actual patches to Linus et al

• Other technologies: working on them ...• Experimentation: Defcon fun :)• Technology Transfer: going well

– LSM in Linux– Immunix licensed to HP

02/07/26 59

Web Resources

• LSM: http://lsm.immunix.org– Active mailing lists

• Sign up if interested

– If you are hacking security into the Linux kernel, consider making it an LSM module

• Defcon:– Defcon convention http://www.defcon.org– CtF game http://www.ghettohackers.net/ctf/