Image processing algorithm Image processing algorithm regression testing frameworkregression testing framework

Soumik Ukil

Testing Ground RulesTesting Ground Rules

Objectives:Objectives:

Testing is the process of executing a program Testing is the process of executing a program with the intent of finding errorswith the intent of finding errors

A good test case is one that has a high A good test case is one that has a high probability of finding an errorprobability of finding an error

A successful test is one that uncovers an errorA successful test is one that uncovers an error

However, testing cannot show the absence However, testing cannot show the absence of defectsof defects

Unit Testing: Testing to determine that individual program

modules perform to specification Basic units of software tested in isolation

Regression Testing: Selective retesting to detect faults introduced

during modification of system Should be performed after any changes in

software like bug fixes

MotivationMotivation

Projects like NETT and BRP Projects like NETT and BRP Complex algorithms which run on hundreds of Complex algorithms which run on hundreds of

datasetsdatasets Source is being constantly modified to fix bugs etcSource is being constantly modified to fix bugs etc Manual testing infeasibleManual testing infeasible

Need automated regression testing Need automated regression testing

Testing StrategiesTesting Strategies

Dynamic AnalysisDynamic Analysis Exercise the software in its execution environmentExercise the software in its execution environment Black BoxBlack Box

Treat the system as black -box, ie no Treat the system as black -box, ie no knowledge of internal structureknowledge of internal structure

Only need to know legal input and expected Only need to know legal input and expected outputoutput

White BoxWhite Box Depends on Internal structure of programDepends on Internal structure of program Test all paths through code Test all paths through code

Testing StrategiesTesting Strategies

Static AnalysisStatic Analysis Code ReviewsCode Reviews WalkthroughsWalkthroughs InspectionsInspections

Tests on image dataTests on image data Gold-standard data used as basis for Gold-standard data used as basis for

correctness of outputcorrectness of output

Black box testsBlack box tests:: Already know 'correct' outputAlready know 'correct' output Generate data from algorithmGenerate data from algorithm Define tests that compare the two sets of dataDefine tests that compare the two sets of data

Tests on image dataTests on image data Generic Tests:Generic Tests:

Check output is of correct datatypeCheck output is of correct datatype Check image dimensionsCheck image dimensions

Specific tests: Specific tests: Depends on application being testedDepends on application being tested For airway tree validation we may want to compare For airway tree validation we may want to compare

distance between branchpointsdistance between branchpoints For Lung Segmentation we need to compare For Lung Segmentation we need to compare

volumes and/or distance between contours volumes and/or distance between contours

Objectives of test frameworkObjectives of test framework

Different applications:Different applications: Regression testing after any changesRegression testing after any changes Validation of dataValidation of data

Flexible:Flexible: Testers only plug in specific testsTesters only plug in specific tests Test data generation and reporting of test results Test data generation and reporting of test results

taken care by frameworktaken care by framework

ImplementationImplementation

Built on top of PyUnitBuilt on top of PyUnit:: Part of standard Python library for Python 2.1 and Part of standard Python library for Python 2.1 and

laterlater

Based on JUnit, a proven testing architectureBased on JUnit, a proven testing architecture

Allows creation of user defined tests, aggregation Allows creation of user defined tests, aggregation into suites, running tests in textual or GUI modeinto suites, running tests in textual or GUI mode

Writing tests with PyUnit:Writing tests with PyUnit:

Basic building blocks called Test CasesBasic building blocks called Test Cases Created by deriving from base class unittest.TestCaseCreated by deriving from base class unittest.TestCase An instance of a TestCase class is an object that can completely run An instance of a TestCase class is an object that can completely run

a single test method a single test method

FixturesFixtures A set-up and tear down method for each test caseA set-up and tear down method for each test case Many different test cases can use the same fixtureMany different test cases can use the same fixture

Test SuitesTest Suites Test case instances can be grouped together according to the Test case instances can be grouped together according to the

features they test features they test All tests can be executed together as part of a suiteAll tests can be executed together as part of a suite

Test RunnerTest Runner A class whose instances run tests and report resultsA class whose instances run tests and report results Can be used in text or GUI modeCan be used in text or GUI mode

Example:Example:class LungSegTestCase(unittest.TestCase):class LungSegTestCase(unittest.TestCase):

def initialize(self,fname1,fname2): def initialize(self,fname1,fname2):

A = AnaFile.AnaFile() # Anafile object for reading imagesA = AnaFile.AnaFile() # Anafile object for reading images

self.data,hdr=A.read(fname1) # segmentation resultself.data,hdr=A.read(fname1) # segmentation resultself.ref_data,ref_hdr = A.read(fname2) # reference maskself.ref_data,ref_hdr = A.read(fname2) # reference mask

def setUp(self):def setUp(self): self.labels=[20,30] # left and right lung labelsself.labels=[20,30] # left and right lung labels

def tearDown(self):def tearDown(self): self.data = Noneself.data = None self.ref_data= Noneself.ref_data= None

def test1(self):## test method names begin 'test*'def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""""""Test to see if mask image datatype is uint8."""

self.assertEquals(self.data.typecode(),'b')self.assertEquals(self.data.typecode(),'b')

def test2(self):def test2(self): """Test to check that correct labels are present in mask file.""""""Test to check that correct labels are present in mask file."""

for n in self.labels:for n in self.labels: errormsg="label" + str(n) + "missing"errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

Example:Example:class LungSegTestCase(unittest.TestCase):class LungSegTestCase(unittest.TestCase):

def initialize(self,fname1,fname2): def initialize(self,fname1,fname2):

A = AnaFile.AnaFile() # Anafile object for reading imagesA = AnaFile.AnaFile() # Anafile object for reading images

self.data,hdr=A.read(fname1) # segmentation resultself.data,hdr=A.read(fname1) # segmentation resultself.ref_data,ref_hdr = A.read(fname2) # reference maskself.ref_data,ref_hdr = A.read(fname2) # reference mask

def setUp(self):def setUp(self): self.labels=[20,30] # left and right lung labelsself.labels=[20,30] # left and right lung labels

def tearDown(self):def tearDown(self): self.data = Noneself.data = None self.ref_data= Noneself.ref_data= None

def test1(self):## test method names begin 'test*'def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""""""Test to see if mask image datatype is uint8."""

self.assertEquals(self.data.typecode(),'b')self.assertEquals(self.data.typecode(),'b')

def test2(self):def test2(self): """Test to check that correct labels are present in mask file.""""""Test to check that correct labels are present in mask file."""

for n in self.labels:for n in self.labels: errormsg="label" + str(n) + "missing"errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

Example:Example:class LungSegTestCase(unittest.TestCase):class LungSegTestCase(unittest.TestCase):

def initialize(self,fname1,fname2): def initialize(self,fname1,fname2):

A = AnaFile.AnaFile() # Anafile object for reading imagesA = AnaFile.AnaFile() # Anafile object for reading images

self.data,hdr=A.read(fname1) # segmentation resultself.data,hdr=A.read(fname1) # segmentation resultself.ref_data,ref_hdr = A.read(fname2) # reference maskself.ref_data,ref_hdr = A.read(fname2) # reference mask

def setUp(self):def setUp(self): self.labels=[20,30] # left and right lung labelsself.labels=[20,30] # left and right lung labels

def tearDown(self):def tearDown(self): self.data = Noneself.data = None self.ref_data= Noneself.ref_data= None

def test1(self):## test method names begin 'test*'def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""""""Test to see if mask image datatype is uint8."""

self.assertEquals(self.data.typecode(),'b')self.assertEquals(self.data.typecode(),'b')

def test2(self):def test2(self): """Test to check that correct labels are present in mask file.""""""Test to check that correct labels are present in mask file."""

for n in self.labels:for n in self.labels: errormsg="label" + str(n) + "missing"errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

Application dependent tests:Application dependent tests:

For lung segmentation we use an area overlap For lung segmentation we use an area overlap measure to test correctness:measure to test correctness:

Check that value is above a given threshold to Check that value is above a given threshold to 'pass' test'pass' test

Similar tests can be defined with distance Similar tests can be defined with distance measures between contoursmeasures between contours

User has to plug-in appropriate testsUser has to plug-in appropriate tests

Flow diagram:Flow diagram:

Sample configuration file:Sample configuration file:[DEFAULT][DEFAULT]# Directory of gold-standard data# Directory of gold-standard dataRefdir = ../ValidationDataRefdir = ../ValidationData

#Directory of data generated by algorithm #Directory of data generated by algorithm Maskdir = ../NewDataMaskdir = ../NewData

# Output file extension# Output file extensionMaskExt = mask.imgMaskExt = mask.img

# Name of logfile directory# Name of logfile directoryLogdir = ../logfilesLogdir = ../logfiles

[ALGORITHM][ALGORITHM]# Dir. of raw image data to run algorithm on # Dir. of raw image data to run algorithm on Inputdir = /home/xyz/lung/DataInputdir = /home/xyz/lung/Data

# Path to executable for algorithm# Path to executable for algorithmExecpath = ../lungExecpath = ../lung

# Input file extension# Input file extensionInputExt = imgInputExt = img

Reporting test results:Reporting test results: Textual output directed to log filesTextual output directed to log files

Success:Success:------------------------------------------------------------------------------------------------------------------------------------------Ran 6 tests in 88.337sRan 6 tests in 88.337sOKOK

Failure:Failure:FAIL: Left lung pixel count test (slice by slice).FAIL: Left lung pixel count test (slice by slice).--------------------------------------------------------------------------------------------------------------------------------------------Traceback (most recent call last):Traceback (most recent call last): File "lungsegtests.py", line 101, in test6File "lungsegtests.py", line 101, in test6 self.failIf(fail==1,errormsg)self.failIf(fail==1,errormsg) File "unittest.py", line 264, in failIfFile "unittest.py", line 264, in failIf if expr: raise self.failureException, msgif expr: raise self.failureException, msgAssertionError: significant segmentation mismatch for left lung on slices:AssertionError: significant segmentation mismatch for left lung on slices:[458, 462][458, 462]

Status:Status:

Testing for Lung segmentation/smoothing has Testing for Lung segmentation/smoothing has been implemented with the following tests:been implemented with the following tests:

Type checking of output masksType checking of output masks Checking that all expected labels are presentChecking that all expected labels are present Slice by slice area comparison for both lungsSlice by slice area comparison for both lungs

Can be used by segmentation algorithms on Can be used by segmentation algorithms on ANALYZE images which produce labeled ANALYZE images which produce labeled masksmasks

Framework can be extended to handle other Framework can be extended to handle other formats like airway tree definitions etcformats like airway tree definitions etc

References:References:

1. http://pyunit.sourceforge.net

2.2. http://www.xprogramming.com/testfram.htmhttp://www.xprogramming.com/testfram.htm

Documentation and source code:

CVS repository: pulmonary/TestFrameWork