Deoendable Software Everywhere

Tao XieAutomated Software Engineering Group

Department of Computer ScienceNorth Carolina State University

https://sites.google.com/site/asergrp/

Automated Software Engineering @NCSU

22

Software Dependability Matters

Loss of Money: Software faults cost the U.S. economy about $59.5 billion each year (0.6% GDP) [NIST 02]

Loss of Life: Faulty medical devices caused 30,000 deaths and 600,000 injuries (1985-2005), with likely 8% due to software faults [FDA 06]

…

33

Improving Software DependabilityTitles of Major Conference Pubs (2005-Present)

4http://people.engr.ncsu.edu/txie/reppubs.html

Improving Software Dependability

Testing & Analysis

Analytics

Reliability

ICSE 12a, ICSE 09aICSE 08, ICSE 05FSE 09, FSE 07ASE 11b, ASE 10, ASE 09aASE 09b, ASE 08a, ASE 07ECOOP 09

ICSE 11, ICSE 10a, ICSE 10bICSE 09b, ICSE 07 FSE 10, FSE 12b, FSE 12cISSTA 11, ISSTA 10, ISSTA 09ASE 11a, ASE 08b, ASE 06OOPSLA 11, ECOOP 06

PerformanceICSE 12bASE 12sp, SRDS 12sp

SIGMETRICS 08

Major Conference Pubs (2005-Present)

10 ICSE, 7 FSE3 ISSTA, 9 ASE3 OOPLSA/ECOOP

Security/PrivacyFSE 11, SIGMETRICS 08WWW 07, ACSAC 08

FSE 12aACSAC 12

Improving Software Dependability

Testing & Analysis

Analytics

Reliability

Performance

ICSE 12a, ICSE 09aICSE 08, ICSE 05FSE 09, FSE 07ASE 11b, ASE 10, ASE 09aASE 09b, ASE 08a, ASE 07ECOOP 09

ICSE 12bASE 12sp, SRDS 12sp

ICSE 11, ICSE 10a, ICSE 10bICSE 09b, ICSE 07 FSE 10, FSE 12b, FSE 12cISSTA 11, ISSTA 10, ISSTA 09ASE 11a, ASE 08b, ASE 06OOPSLA 11, ECOOP 06

FSE 11, SIGMETRICS 08WWW 07, ACSAC 08

SIGMETRICS 08

Major Conference Pubs (2005-Present)

6

10 ICSE, 7 FSE3 ISSTA, 9 ASE3 OOPLSA/ECOOP

Security/PrivacyFSE 11, SIGMETRICS 08WWW 07

Artifacts Under Analysis• DB apps• GUI apps• Web/SOA apps•Mobile apps• Cloud apps• Search engines•AC/Firewall policies

• API docs• Bug reports• Requirements doc• Execution traces• …

FSE 12a

Microsoft Research Pex Incubation Project for Visual Studio

Download counts (20 months)(Feb. 2008 - Oct. 2009 )

Academic: 17,366 Devlabs: 13,022 Total: 30,388

The contributed Fitnex search strategy [DSN 2009] included in Pex releases since Sept. 2008

7

http://research.microsoft.com/en-us/projects/pex/

Loops Fitnex [DSN 09]

Environments File system apps [AST 09] Database apps [ASE 10-sp, ASE 11] Cloud apps [IEEE Software 12]

Method sequences Seeker [OOPSLA 11], MSeqGen [ESEC/FSE 09]

Opportunities Regression testing [ISSTA 11] Developer guidance (cooperative developer testing)

[ICSE 12]

Challenges of Dynamic Symbolic Execution

http://research.microsoft.com/en-us/projects/pex/community.aspx#publications

Microsoft Research Pex for FunTeaching and Learning CS via Social Gaming

1,013,336 clicked 'Ask Pex!'

www.pexforfun.com

The contributed concept of Coding Duel games as major game type of Pex for Fun since Summer 2010

9http://www.pexforfun.com/

Behind the Scene of Pex for Fun

Secret Implementation class Secret {

public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation class Player {

public static int Puzzle(int x) { return x; }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

10

Coding Duel Competition@ICSE 2011

Microsoft Research AsiaSoftware Analytics

Recent and ongoing work (e.g., StackMine [ICSE 12b], XIAO [ACSAC 12]) with successful technology transfer in collaboration with Microsoft Research Asia

12http://research.microsoft.com/en-us/groups/sa/

StackMine

Performance debugging in the large via mining millions of stack traces

[ICSE 2012]

http://people.engr.ncsu.edu/txie/publications.htm#icse12-stackmine

ICSE 2012 14

Performance debugging in the large

Pattern Matching

Trace StorageTrace

collection

Bug update

Problematic Pattern

RepositoryBug

DatabaseNetwork

Trace analysis

How many issues are still unknown?

Which trace file should I investigate

first?

Bug filing

Key to issue discoveryBottleneck

of scalability

ICSE 2012 15

Impact“We believe that the MSRA tool is highly valuable and much more efficient for mass trace (100+ traces) analysis. For 1000 traces, we believe the tool saves us 4-6 weeks of time to create new signatures, which is quite a significant productivity boost.”

Highly effective new issue discovery on Windows mini-hang

Continuous impact on future Windows versions

XIAOScalable code clone analysis

[ACSAC 2012]

http://people.engr.ncsu.edu/txie/publications.htm#acsac12-xiao

ICSE 2012 17

XIAO: Code Clone Analysis

Motivation Copy-and-paste is a common developer

behavior A real tool widely adopted internally and

externally XIAO enables code clone analysis in the

following way High tunability High scalability High compatibility High explorability

ICSE 2012 18

Benefiting developer community

Available in Visual Studio 2012 RC

Searching similar snippets for fixing bug

onceFinding refactoring

opportunity

ICSE 2012 19

More secure Microsoft products

Code Clone Search service integrated into workflow of Microsoft Security Response Center

Over 590 million lines of code indexed across multiple products

Real security issues proactively identified and addressed

20

Example – MS Security Bulletin MS12-034

MSRC: Microsoft Security Response Center

Combined Security Update for Microsoft Office, Windows, .NET Framework, and Silverlight, published: Tuesday, May 08, 20123 publicly disclosed vulnerabilities and seven privately reported involved. Specifically, one is exploited by the Duqu malware to execute arbitrary code when a user opened a malicious Office document

Insufficient bounds check within the font parsing subsystem of win32k.sysCloned copy in gdiplus.dll, ogl.dll (office), Silver Light, Windows Journal viewerMicrosoft Technet Blog about this bulletinHowever, we wanted to be sure to address the vulnerable code wherever it appeared across the Microsoft code base. To that end, we have been working with Microsoft Research to develop a “Cloned Code Detection” system that we can run for every MSRC case to find any instance of the vulnerable code in any shipping product. This system is the one that found several of the copies of CVE-2011-3402 that we are now addressing with MS12-034.

Government AgencyNIST

Jointly-developed ACPT (Access Control Policy Tool) beta release being beta-tested in ~130 organizations/users

http://csrc.nist.gov/groups/SNS/acpt/index.html

Government Agencies FDA

Test a point-of-care assistant medical device [ASE 10sp] and mine FDA incident reports

Our PhD student Rahul Pandita, part of the FDA Semantic Data Mining Development Team, co-recognized with 2012 FDA Group Recognition Award

23

Mining Textual Software Artifacts• Detect duplicate bug reports [ICSE 08]• Identify security bug reports [MSR 10]• Mine resource specs from Javadoc [ASE 09, Best

Paper Award]• Mine code contracts from API docs [ICSE 12]• Mine security policies from requirements

docs [FSE 12]javax.resource.cci.Connection

createInteraction():“Creates an interaction associated with this connection.”getMetaData():“Gets the information on the underlying EIS instance represented through an active connection.”close():“Initiates close of the connection handle at the application level.”

Various countries/regions Software internationalization▪ Locating constant strings to translate [ICSE 09,

FSE 10]▪ E.g., translating Megamek (a realtime strategy

game)

Various programming languages PL translation▪ E.g., translating Java to C# [ICSE 10]

Dependable Software Everywhere

Various types of software Database applications [ASE 10-sp] Network/file-system applications [AST 09] Game applications [ICSE 09] Cyber-physical systems (power grid,

medical device software, …) Mobile/could applications Social network applications …

Dependable Software Everywhere

TouchDevelop @MSRMobile application development environmento create applications

(aka “scripts”) on the phone itself

o no PC requiredo access to phone

sensors, camera, music, web, etc.

o share scripts with other people

o After 9 months, > 6000 games/apps written and published by users

NCSU ASE Group: Source code license to analysis infrastructure

http://research.microsoft.com/projects/touchdevelop/

Teaching for students

engaging experience work with your personal data (pictures,

songs, …) create games on the go

lowers bar of entry to programming create tailored apps for micro-business

Nikolai Tillmann, Michal Moskal, Jonathan de Halleux, Manuel Fahndrich, Judith Bishop, Arjmand Samuel, and Tao Xie. The Future of Teaching Programming is on Mobile Devices. In Proceedings of 17th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2012), 2012.

Language+Editor typed language

enables precise auto-completion suggestions

imperative object-oriented, but doesn’t

allow definition of new objects semi-structured editor

structured at statement level unstructured at expression level structured at token level

Rich APIs

All phone features are available via TouchDevelop: camera, touch, accelerometer, compass,

gyro, microphone, … contacts, calendar, email, … pictures, songs, … web queries, search, maps, social

networks graphics with physics engine tiles

Script SharingScript bazaar in cloud:central authority for script sharing backup anyone can publish a script script source code

is made available discover new scripts! all published scripts are

analyzed (privacy) Xusheng Xiao’s internship work!Xusheng Xiao, Nikolai Tillmann, Manuel

Fahndrich, Jonathan de Halleux, and Michal Moskal, User-Aware Privacy Control via Extended Static-Information-Flow Analysis. In Proc. ASE 2012

Drustworthy Software Everywhere

http://people.engr.ncsu.edu/txie/https://sites.google.com/site/asergrp/