AppSensor Near Real-Time Event Detection and Response - DevNexus 2016

AppSensor~real-time event detection and response

me• appsensor dev lead (OWASP)

• dev / security

!

• twitter: @_jtmelton

• email: jtmelton at gmail.com

• github: jtmelton

agenda

• history (recent)

• motivations / problems

• solution / tech

• future / wrap-up

thesis: !

modern secure applications protect themselves against

attackers

history

~5 yrs ago dev

• mostly web apps [RoR, PHP, .NET, Java)

• ajax (jquery) use growing

• mobile just getting started

• deployment to VMs

• hadoop picking up

• BI tools

• AWS starting

• cloud hype cycle (NIST defines)

~now dev• JS everywhere

• functional / rx programming

• cloud everything

• ci/cd

• nosql / CAP light

• containers

• big data

• stream processing

• config management

• iot

• beacons [usage, ads, errors, performance]

• actors/csp

• microservices

• cqrs / event sourcing

• mobile

~now dev• JS everywhere

• functional / rx programming

• cloud everything

• ci/cd

• nosql / CAP light

• containers

• big data

• stream processing

• config management

• iot

• beacons [usage, ads, errors, performance]

• actors/csp

• microservices

• cqrs / event sourcing

• mobile 1 .. * of [scale, speed, cloud, lack of environmental access]

(brief) history

- LinkedIn, March 2015

“the Kafka ecosystem at LinkedIn is sent over 800 billion* messages per day..

At the busiest times of day, we are receiving over 13 million messages per second.”

* Update (Sept 2015) : 1.1 Trillion messages per day

last ~5 yrs security

• 3rd party libs (dep-check)

• bug bounties

• sast / dast evolve (ZAP)

• iast / rasp

• http security headers

• automatic encoding (JXT)

• *-monkey -NetflixOSS

• bdd-security/gauntlt

• ci/cd plugins

• 2fa

• osquery

1 .. * of [scale, speed, cloud, lack of environmental access]

dev vs. security

• dev is exploiting fundamental architectural and deployment changes to add business value

!

• security is iterating on existing solutions - and - trying to close gaps (known problems)

security ~5 yrs ago vs today

?

motivations

“security”

• confidentiality and integrity important

• availability often ignored by security (informs the whole industry- eg. tooling)

• if availability important, runtime important

http://i.imgur.com/TdtgvtW.png

your environment

• how many concurrent users do you have right now?

• what are your users doing in the app?

https://github.com/aphyr/jepsen-talks/blob/master/2015/goto/goto.pdf

Security defects are a subset of all defects

catching defects

• what do dev/qa do for functionality?

• test [unit, integration, system, manual, tools]

• what do attackers do for security?

• test [automated tools, manual]

observations

• attackers do bad things

• bad things often easily recognizable (to you)

• attacker success often* requires > 1 attempt

* If not, you lose

robbing a bankPhysical Controls

Electronic Monitoring

Human Monitoring

Instant Detection and Response

Controlled Access

Multi Factor Auth

Transaction Verification

would you bank here?

Alternate Admin Access

Partial External Controls

Ineffective Monitoring

No Real Time Analysis

Unnecessary Partner Trust No Response

Capability

Single Factor Auth

Limited Security Training

http://worth1000.s3.amazonaws.com/submissions/414000/414200_9830_1024x2000.jpg

defender’s dilemma

• attacker needs ONE successful attack

• defender must defend ALL attacks

!

• NOTE: you are defenders

on people

• 18.2 million devs

• 200K security (all, not appsec only)

• ~ 1.1 sec : 100 dev

!

• 1.75 sec : 100 dev (bsimm)

security modern dev

• polyglot static and dynamic languages

• microservices / soa

• json, thrift, protobuf, etc. endpoints

• WebAssembly ???

• a single mature, static language

• monolith

• http (really html) endpoints

in summary …• “traditional” security, dev, ops doesn’t know what’s going

on in the app at runtime

• security defects exist

• attackers don’t magically know what’s vulnerable

• existing “monitoring” usually terrible

• there will never be enough “security” people

• “traditional” security tooling doesn’t fit modern dev

• actual defense is _really_ hard

the pitch !

(a humble proposal)

having to deal with [scale, speed, cloud, lack of

environmental access].. !

..this as of now incomplete transition..

!

..represents an enormous opportunity for security

the pitch• in addition to a secure SDLC … (ie. > 1 request/attack)

!

• figure out what’s happening at runtime

• make intrusion detection primitives available in app

• exploit automated response > manual response

• stop attacker before success

• get self-protecting applications and valuable intel

X successAppSensor

X

prior work

• Network IDS (Denning, * others, NIST SP800-94)

• Intrusion prevention

• Fraud detection

• Rules engines, Risk analysis/reduction, HIDS

terminology

• event - suspicious

• attack - malicious (1 .. * events)

• response - take action (1 .. 1 attack)

• detection point - activity category (e.g. cookie modification)

the tech

Architecture

• picture 1

Correlation

Adding Detection Points

adding detection points

• manually

• reverse proxy

• owasp aside

manualPOST /account/transfer HTTP/1.1 !Host: 127.0.0.1 User-Agent: Mozilla/5.0 (Win…) Accept: text/html,application/xhtml+xml Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Referer: http://127.0.0.1/account.php Cookie: PHPSESSID=l9…lgt5 Connection: keep-alive Content-Type: application/x-www-form-urlencoded Content-Length: 30 from_acct=xxx1234&to_acct=xxx9876&amt=20.00

manualPOST /account/transfer HTTP/1.1 !Host: 127.0.0.1 User-Agent: Mozilla/5.0 (Win…) Accept: text/html,application/xhtml+xml Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Referer: http://127.0.0.1/account.php Cookie: PHPSESSID=l9…lgt5 Connection: keep-alive Content-Type: application/x-www-form-urlencoded Content-Length: 30 from_acct=xxx1234&to_acct=xxx9876&amt=20.00

manualPOST /account/transfer HTTP/1.1 !Host: 127.0.0.1 User-Agent: Mozilla/5.0 (Win…) Accept: text/html,application/xhtml+xml Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Referer: http://127.0.0.1/account.php Cookie: PHPSESSID=l9…lgt5 Connection: keep-alive Content-Type: application/x-www-form-urlencoded Content-Length: 30 from_acct=xxx1234&to_acct=xxx9876&amt=20.00

manual@POST public Response transfer( String from, String to, String amount) { ! transfer(from, to, amount); ! return Response.ok(); }

manual@POST public Response transfer( String from, String to, String amount) { ! if ( currentUser.owns(from) ) { transfer(from, to, amount); } ! return Response.ok(); }

manual@POST public Response transfer( String from, String to, String amount) { ! if ( currentUser.owns(from) ) { transfer(from, to, amount); } else {! showErrorPage();! // normal error handling! }!! return Response.ok(); }

manual@POST public Response transfer( String from, String to, String amount) { ! if ( currentUser.owns(from) ) { transfer(from, to, amount); } else {!! appsensor.addEvent( new Event(currentUser, "ACE2") );! showErrorPage(); // normal error handling! }!! return Response.ok(); }

appsensor-reverse-proxy

appsensor-reverse-proxy

• written in go

• blocks requests

• canned detection points (toggle-able)

• easily extendable

• https://github.com/jtmelton/appsensor-reverse-proxy

OWASP ASIDE

• UNCC SIS project

• secure programming IDE plugin

• educational component

• https://www.owasp.org/index.php/OWASP_ASIDE_Project

OWASP ASIDE

• eclipse IDE

• reminder icon or highlight

• drop down list of applicable sensors

• auto-insertion of ASIDE sensor APIs and code refactoring

OWASP ASIDE

OWASP ASIDEBased  on  ESAPI  code  (length  checked),  ASIDE  infers  that  this  may  be  a  point  to  insert  an  app  sensor;  whether  a  sensor  is  placed  relies  on  developer’s  decision.  

OWASP ASIDE

It  not  only  captures  the  context  informaFon  (e.g.  the  sensor  event  is  from  username  field),  but  also  records  that  the  sensor  event  is  due  to  an  exceedingly  lengthy  input.  

Detec%on(Point(Type( Detec%on(Points(Covered(

Authen'ca'onExcep'on. AE4:.Unexpected.Quan'ty.of.Characters.in.Username.AE5:.Unexpected.Quan'ty.of.Characters.in.Password.AE6:.Unexpected.Type.of.Character.in.Username.AE7:.Unexpected.Type.of.Character.in.Password.

InputExcep'on. IE1:.Cross.Site.Scrip'ng.AEempt.

EncodingExcep'on. EE1:.Double.Encoded.Character.EE2:.Unexpected.Encoding.Used.

CommandInjec'onExcep'on.

CIE1:.Blacklist.Inspec'on.for.Common.SQL.Injec'on.Values.

Detec%on(Points(Picked( Corresponding(ASIDE(APIs(

AE4:%Unexpected%Quan1ty%of%Characters%in%Username%AE5:%Unexpected%Quan1ty%of%Characters%in%Password%

Java.lang.String%ASIDE.Quan%tyExcep%onSensor(Java.lang.String%parameter)%

AE6:%Unexpected%Type%of%Character%in%Username%AE7:%Unexpected%Type%of%Character%in%Password%

Java.lang.String%ASIDE.TypeExcep%onSensor(Java.lang.String%parameter)%

IE1:%Cross%Site%Scrip1ng%AKempt%

Java.lang.String%ASIDE.XSSSensor(Java.lang.String%parameter)%

EE1:%Double%Encoded%Character%EE2:%Unexpected%Encoding%Used%

Java.lang.String%ASIDE.EncodingExcep%onSensor(Java.lang.String%parameter)%

CIE1:%Blacklist%Inspec1on%for%Common%SQL%Injec1on%Values%

Java.lang.String%ASIDE.SQLInjec%onSensor(Java.lang.String%parameter)%

Viewing Data

owasp SoC sprint

• Sumanth Damarla

• appsensor -> ELK stack

• appsensor -> influxdb -> grafana

owasp SoC sprint

owasp SoC sprint

machine learning

• very simple analysis

• generated demo dataset for 1-week

• build base model

• look for “anomalies”

appsensor-ui

• spring boot + reactjs

• simple dashboards / trends

future plans

future• reverse proxy

• appsensor-ui

• framework integration for detection points (spring security exists, add others)

• analysis engines - 2 grad students (David Scrobonia, Kelvin Brown Bomah) doing projects here (rules and ML)

• ???

you• help wanted!

• plenty of places to contribute and improve

• friendly, helpful community

• https://github.com/jtmelton/appsensor/issues

• https://www.owasp.org/index.php/OWASP_AppSensor_Project#tab=Road_Map_and_Getting_Involved

wrap-up

related projects

• repsheet

• ensnare

• fido

• riemann

• elastalert

pick a tool (or 2) … !

but use the idea

contributors• https://www.owasp.org/index.php/

OWASP_AppSensor_Project#tab=Acknowledgements

links

• https://www.owasp.org/index.php/OWASP_AppSensor_Project

• http://appsensor.org/

• https://github.com/jtmelton/appsensor

?