1

Proposing SQL Statement Coverage Metrics

Ben Smith

Yonghee Shin

Laurie Williams

2

Agenda

• Motivation and Objective

• Testing for Security

• SQL Injection Vulnerabilities/Attacks

• Terms and Metrics

• Case Study

• Limitations

• Future Work

3

Cyber Vulnerabilities (1995-2007)

0

1000

2000

3000

4000

5000

6000

7000

8000

1995 1997 1999 2001 2003 2005 2007

Year

Re

po

rte

d V

uln

era

bili

tie

s

4

Cyber Vulnerabilities, cont’d

• Averages from NVD 2003 – 2007 [1]

• Input validation vulnerabilities comprise >50% of reported total

• SQL Injection Vulnerabilities comprise 11% of reported total

5

Objective

To propose coverage metrics which determine the adequacy of a test set’s ability to mitigate SQL injection attacks.

6

Security Testing [2]

Intended Functionality (Requirements Specification)

Application as Coded

Unintended Functionality

Missing or Incorrect Functionality

7

$username = $_POST[‘username’];$password = $_POST[‘password’];

$result = mysql_query(“select * from users where username =‘’ OR 1=1 ---’ AND password = ‘$password’”);

$firstresult = mysql_fetch_array($result);

$role = $firstresult[‘role’];

$_COOKIE[‘userrole’] = $role

SQL Injection Attacks

$username = $_POST[‘username’];$password = $_POST[‘password’];

$result = mysql_query(“select * from users where username =‘$username’ AND password = ‘$password’”);

$firstresult = mysql_fetch_array($result);

$role = $firstresult[‘role’];

$_COOKIE[‘userrole’] = $role;

‘ OR 1=1 --

8

Web Application Security

InternetInternetData

Web App

Query

Query

9

Coverage Analysis

10

Theory

Higher SQL Coverage Metrics Higher System

Security

11

Research Plan

Blacklist/Input Validation Testing

SQL Coverage Metrics

12

Terms and Metrics (1)

Target Statement: a SQL statement which could cause a security problem when malicious input is used, for example:

java.sql.Connection conn = factory.getConnection();java.sql.PreparedStatement ps =

conn.prepareStatement("UPDATE globalVariables SET value = ? WHERE name = ?;");

ps.setInt(1, value);ps.setString(2, name);java.sql.ResultSet rs = ps.executeQuery();

13

Terms and Metrics (2)

Input Variable: any variable in the server-side production code which is dynamically user-assigned and sent to the database management system. For example,

java.sql.Connection conn = factory.getConnection();java.sql.PreparedStatement ps =

conn.prepareStatement("UPDATE globalVariables SET value = ? WHERE name = ?;");

ps.setInt(1, value);ps.setString(2, name);java.sql.ResultSet rs = ps.executeQuery();

14

Terms and Metrics (3)

Target Statement Coverage: the percentage of target statements in the production code executed/covered at least once by the test set.

Input Variable Coverage: the percentage of input variables in the production code executed/covered at least once by the test set.

15

Case Study: iTrust [3]

• Role-based web healthcare application

• Developed by NC State students in a security testing class

• Aims to be HIPAA-compliant

• ~7700 SLOC, 143 production classes

• Tomcat + MySQL• 84% line coverage

16

Case Study: Overview

• Manually instrument iTrust to record the first execution of each target statement

• Run the built-in unit tests to reveal the coverage result for both SQL statement coverage and input variable coverage

• Warning: Formative Research!!

17

Case Study: Results

Coverage Metric Tested Total Result

Target Statement 90 93 96.7%

Input Variable 209 212 98.5%

18

A few more results…

• Three-tiered web applications

• SQL

• Tomcat

• Provided test sets

• Examples– Hispacta– IceScrum– Care2x?

19

A few more results…

iTrust IceScrum Hispacta

Type of Tests Run Unit Unit Unit + Integration

Lines of Code 7,707 19,442 1,991

Production Classes 143 155 42

DAO Classes 20 10 4

SQL Statements 93 96 23

Covered by Tests 90 29 18

Coverage 96.7% 30.2% 78.3%

SQL Input Variables 212 517 36

Covered by Tests 209 258 32

Coverage 98.5% 49.9% 88.9%

Line Coverage (EclEmma)

84.1% 9.7% 49.4%

20

Threats to Validity

• Human error

• Inaccurate analysis

• May not be feasible in other systems, languages or operational environments

• iTrust is highly testable, and well-tested

21

Future Work

• Empirical studies– SQL coverage in other open source systems– Linking SQL coverage to higher security levels

• Automation– Dynamic Queries– Identification

• Finer-grained metrics

• Use as platform for blacklist/whitelist testing

22

Very Future Work

Blacklist/Input Validation Testing

SQL Coverage Metrics

A Framework

23

References

[1] United States National Vulnerability Database. http://nvd.nist.gov

[2] H. H. Thompson, "Why security testing is hard," Security & Privacy Magazine, IEEE, vol. 1, no. 4, pp. 83-86, 2003.

[3] iTrust website. http://sourceforge.net/projects/iTrust

24

•Questions?

•Feedback & Comments Welcome!

25

Dynamic Queriesjava.sql.Connection conn = factory.getConnection();java.sql.PreparedStatement ps = null;

if (user.isAdmin()){

ps = conn.prepareStatement(“select * from adminTable;");

}else{

ps = conn.prepareStatement(“select * from generalTable;”);

}

ps.executeQuery();

26

Case Study: Instrumentation

java.sql.Connection conn =factory.getConnection();java.sql.PreparedStatement ps =conn.prepareStatement("UPDATEglobalVariables set SET VALUE = ? WHEREName = ‘Timeout’;");ps.setInt(1, mins);SQLMarker.mark(1, 1); java.sql.ResultSet rs = ps.executeQuery();