DataModelAnalysisTevfikBultan

UniversityofCaliforniaSantaBarbara

Jointworkwith

Jaideep Nijjar andIvanBocic

WebApplicationDependability

2

WebApplicationDependability

3

WebApplicationDependability

President Obama: “I want to go in and fix myself, but I don't write code"

• TRACKS:Atodolistapplication

WebApplicationDependability

5

Context Recurring Todo

Feed the Dog EDIT

WebApplicationArchitecture

• ModelViewController(MVC)pattern:RubyonRails,Zend forPHP,CakePHP,StrutsforJava,Django forPython,…• ObjectRelationalMapping(ORM)ActiveRecord,Hibernate,…

6

RESTful Controller

View

OOP Rel DbORM

DataModel

AnExampleRailsDataModel

7

class User < ActiveRecord::Base

has_many :todos

has_many :projects

end

class Project < ActiveRecord::Base

belongs_to :user

has_many :todos

has_many :notes

end

class Todo < ActiveRecord::Base

belongs_to :user

belongs_to :project

end

class Note < ActiveRecord::Base

belongs_to :project

end

class ProjectsController < ApplicationController

def destroy

@project = Project.find(params[:project_id])

@project.notes.each do |note|

note.delete

end

@project.delete

respond_to(...)

end

end

Static Data Model

Data Model Updates: Actions

StaticDataModel

• ActiveRecord classdeclarations• setsofobjects

• ActiveRecord associationdeclarations• has_one,has_many,belongs_to,has_and_belongs_to_many

• Associationdeclarationscanbeusedtodeclarethethreebasictypesofrelationsbetweenclasses• one-to-one• one-to-many• many-to-many 8

ExtensionstoStaticDataModel• :throughOption

• Toexpressrelationswhicharecompositionofotherrelations

• :conditionsOption• Torelateasubsetofobjectstoanotherclass

• :polymorphicOption• Toexpresspolymorphicrelationships

• :dependentOption• Ondelete,thisoptionexpresseswhethertodeletetheassociatedobjectsornot

9

The:throughOptionclass User < ActiveRecord::Base

has_one :profile

has_many :photos, :through => :profileend

class Profile < ActiveRecord::Base

belongs_to :user

has_many :photos

end

class Photo < ActiveRecord::Base

belongs_to :profile

endProfile

User Photo

*

*

1 1

1

1

10

The:dependentOption

• :delete directlydeletetheassociatedobjectswithoutlookingatitsdependencies

• :destroy firstcheckswhethertheassociatedobjectsthemselveshaveassociationswiththe:dependent optionset

class User < ActiveRecord::Basehas_one :profile, :dependent => :destroy

end

class Profile < ActiveRecord::Basebelongs_to :userhas_many :photos, :dependent => :destroy

end

PhotoProfileUser *1 11

11

DataModelVerification

• Formalizethestaticdatamodelas• Asetofclasses• A setofrelationsbetweenthoseclasses• Asetofconstraintsontherelationsthatareimposedbytheassociationdeclarations

• Givenaformaldatamodelwecanautomaticallycheckifagivenpropertyholdsforthedatamodel• Automatedverificationdetermines:Dotheconstraintsofthedatamodelimplytheproperty?

12

DataModelVerification

AlloyEncoder

instanceorunsat

formula

formaldatamodel+property

AlloyAnalyzer

Property

ActiveRecord

SMTSolver

instanceorunsatorunknown

formulaSMT-LIBEncoder

PropertyFailed+Counterexample

PropertyVerified

Unknown

ModelExtraction

ResultsInterpreter

ResultsInterpreter

BOUNDED VERIFICATION UNBOUNDED VERIFICATION

nBound

bound

13

HowAutomatedisAutomatedVerification?• Allexceptonestep:Propertyspecification• Example:ItispossibletohaveaUserwhodoesnothaveanyPhotos.• InAlloy:

pred prop{alls:PreState |someu:User|allp:Photo|

• InSMT-LIB:

• Canwemakeiteasier?14

(pnotin(s.photo_user).u)}

(assert(exists((aPolymorphicClass))(forall ((pPhoto))(and(isUser a)(not(=p(auser_photop)))))))

PropertyTemplates

• Propertytemplatesforpropertyspecification• Language-neutral• DonotrequirefamiliaritywithSMT-LIBandAlloy

• Examplepropertytemplate:• noOrphans[classA,classB]

• Tocheckthatdeleting anobject fromclassA does notcause relatedobjects inclassB tobeorphaned

• Easilyreruntoolandswitchtheverificationtechnique,withouthavingtorewritetheproperty

• Wedevelopedsevenpropertytemplatesforthemostcommondatamodelproperties

15

CanWeDoMore?

AutomaticPropertyInference

• Automaticallyinferpropertiesbasedondatamodelschema• Datamodelschema:Adirected,annotatedgraphthatrepresentstherelations

• Lookforpatternsinthedatamodelschemaandinferapropertyifapatternthatcorrespondstoapropertyappears

• Forexample,orphanprevention

17

0 1 n. . .

. . .

CanWeDoEvenMore?

• noOrphans(X,Y)propertyfailingmeansdeletinganobjectfromclassXcreatesanorphanchainthatstartswithassociatedobjectinclassY

• Repair:Set:dependent optionto:destroy onassociationdeclarationinclassXandonremainingrelationsinthechainthatstartswithclassY

Set :dependent => :destroy on all relations in chain

X Y N. . .

. . .

18

AutomatedDataModelRepair

Summary

ModelExtraction

Verification

VerificationResults

ActiveRecords

FormalDataModel+Properties

PropertyInference

FormalDataModel

19

DataModelRepair

for failingproperties

ExperimentResultsApplication PropertyType #Inferred #Timeout #Failed

LovdByLessdeletePropagates 13 0 10

noOrphans 0 0 0

transitive 1 0 1

SubstructdeletePropagates 27 0 16

noOrphans 2 0 1

transitive 4 0 4

TracksdeletePropagates 15 0 6

noOrphans 1 0 1

transitive 12 0 12

FatFreeCRMdeletePropagates 32 1 19

noOrphans 5 0 0

transitive 6 2 6

OSRdeletePropagates 19 0 12

noOrphans 1 0 1

transitive 7 0 7

TOTAL 145 3 96

20

PropertyType#DataModel&Application

Errors

#DataModelErrors

#Failures DuetoRails

Limitations

#FalsePositives

deletePropagates 1 9 0 0

noOrphans 0 0 0 0

transitive 0 0 0 1

deletePropagates 1 3 5 7

noOrphans 0 1 0 0

transitive 0 1 0 3

deletePropagates 1 1 3 1

noOrphans 0 0 0 1

transitive 0 7 0 5

deletePropagates 0 18 1 0

noOrphans 0 0 0 0

transitive 0 0 0 6

deletePropagates 0 12 0 0

noOrphans 0 1 0 0

transitive 0 7 0 0

TOTAL 3 60 9 28

21

WhatAboutDataModelActions?

22

class User < ActiveRecord::Base

has_many :todos

has_many :projects

end

class Project < ActiveRecord::Base

belongs_to :user

has_many :todos

has_many :notes

end

class Todo < ActiveRecord::Base

belongs_to :user

belongs_to :project

end

class Note < ActiveRecord::Base

belongs_to :project

end

class ProjectsController < ApplicationController

def destroy

@project = Project.find(params[:project_id])

@project.notes.each do |note|

note.delete

end

@project.delete

respond_to(...)

end

end

Static Data Model

Data Model Updates: Actions

VerificationofDataModelActions

23

AbstractDataStores

24

class Userhas_many :todoshas_many :projects

end

class Projectbelongs_to :userhas_many :todoshas_many :notes

end

class Todobelongs_to :userbelongs_to :project

end

class Notebelongs_to :project

end

class User {0+ Todo todos inverseof user0+ Project projects inverseofuser

}

class Project {0..1 User user0+ Todo todos inverseof project0+ Note notes inverseof project

}

class Todo {0..1 User user0..1 Project project

}

class Note {0..1 Project project

}

Rails Abstract Data Store

AbstractDataStores

25

def project_destroy@project = Project.find(

params[:project_id])@project.notes.each do |note|

note.deleteend@project.deleterespond_to(...)

end

action project_destroy() {at_project =

oneof(allof(Project))foreach note: at_project.notes {

delete note}delete at_project

}

invariant(forall{ |project|!project.user.empty?

})invariant(forall{ |user|

user.projects.todos.include?(user)})

forall(Project project:not empty(project.user)

)forall(User user:

user in user.projects.todos.users)

Our library allows developers to specify invariants in native Ruby

Extraction

Extractionishardforactions• Dynamictypesystem• Metaprogramming• Eval• GhostMethodssuchas: User.find_by_name(‘Rob’)

Observations• Theschemaisstatic• Actiondeclarationsarestatic• ORMclassesandmethodsdonotchangetheirsemantic

duringexecution• eveniftheimplementationcodeisgenerateddynamically

26

ExtractionviaInstrumentedExecution

• Boot-uptheRailsruntimeinasimulatedenvironment• Withoutopeningsocketsorconnectingtothedatabase

• Prepareactionmethodsforextraction• ORMoperationswillrecordtheirinvocationinsteadof

communicatingwiththedatabase• Methodcallspropagateinstrumentationjustbefore

execution• Extractionispathinsensitive,executingbothbranches

subsequently

• TriggeranHTTPrequestthattriggersanaction27

VerificationviaTranslationtoFOL

• ApredicateisgeneratedforeachclassandassociationUser(o) meansthato isaninstanceofUserProject_user(t) meansthatt representsanassociationbetweenaProjectobjectandUserobject

• Typesystemconstraintsbecomeaxioms∀u:User(u)→¬(Project(u)∨ Todo(u)...)

• Cardinalityofassociationsisexpressedthroughaxiomseg.0..1:∀t1,t2:(Project_user(t1)∧ Project_user(t2)∧

Project_user_lhs(t1)=Project_user_lhs(t2))→Project_user_rhs(t1)=Project_user_rhs(t2)

28

TranslationofStatementstoFOL

• Anactionisasequentialcompositionofstatements.

• Statements• Astateisrepresentedwithapredicatedenotingallentities

thatexistinastate• Astatementisamigrationbetweenstates

e.g.,acreate Note statement:¬pre_state(newly_created())¬∃t:post_state(t)∧ Note_project_lhs(t)=newly_created()∀o:(post_state(o)↔(pre_state(o)∨ o=newly_created())) 29

TranslationofLoopstoFOL

• WeonlysupportForEach loops(fornow)• Theycorrespondtouniversalquantification

• Statementscanexecutemultipletimesinloops• Contextstodifferentiateiterations

• Orderingofiterations• Iterationinterdependence

30

InductiveVerification

• Inv(s) isaformuladenotingthatallinvariantsholdinstates

• Action(s,s’) isaformuladenotingthattheactionmaytransitionfromstates tostates’

Checkif: ∀s,s’:Inv(s)∧ Action(s,s’)→Inv(s’)

31

Experiments

Experimentedon3 opensourceRailsapplications• FatFreeCRM,Tracks,Kandan• 272 actions, 23 invariants

Identified4 bugs• Reportedtooriginaldevelopers• Allimmediatelyconfirmedand,since,fixed• Missedbypreviousverificationeffortsontheseapplications

32

Experiments

33

Publications•JaideepNijjar andTevfikBultan.BoundedVerificationofRubyonRailsDataModels.InProc.InternationalSymposiumonSoftwareTestingandAnalysis(ISSTA),pages67–77,2011.•JaideepNijjar andTevfikBultan.UnboundedDataModelVerificationUsingSMTSolvers.InProc.27thIEEE/ACMInt.Conf.AutomatedSoftwareEngineering(ASE),pages210–219,2012.•JaideepNijjar,IvanBocic andTevfikBultan.AnIntegratedDataModelVerifierwithPropertyTemplates.In Proc.1stFMEWorkshoponFormalMethodsinSoftwareEngineering(FormaliSE 2013).•JaideepNijjar andTevfikBultan.DataModelPropertyInferenceandRepair.InProc.InternationalSymposiumonSoftwareTestingandAnalysis(ISSTA),pages202—212,2013.•IvanBocic,andTevfikBultan.InductiveVerificationofDataModelInvariantsforWebApplications.InProc.InternationalConferenceonSoftwareEngineering(ICSE),2014•JaideepNijjar,IvanBocic,andTevfikBultan.DataModelPropertyInference,VerificationandRepairforWebApplications.(SubmittedtoACMTranslationsonSoftwareEngineeringandMethodology).

34