Upload
aiacov
View
3.288
Download
9
Tags:
Embed Size (px)
DESCRIPTION
What is Sikuli? How it works? How to use it?
Citation preview
Automation with Sikuli
AgendaWhat you’ll learn today
What is Sikuli?
How it works?
How to use it?
What is Sikuli ?
Visual approach to search and automation of graphical user interfaces using screenshots.
Motivation
Usually needed to automate GUIs: support from developers API access language/OS dependency position/naming dependencies
Just to see if it can be done
Demo
System design
Finding GUI patterns on the screen
Invariance
Resized versions
Texturally similar, different color pallets
Template Matching for small patterns
Invariant local features for big patterns
Learn Extract model from training pattern▪ Invariant to scale & rotation
Encode in the model the relative position of the center
Search Extract invariant features Infer possible model center position Cluster consistent features Validate supposition
Find visual objects
find( )
Finder
f = Finder(path-to-imagefile)
f.find(path-to-imagefile, [similarity])
// iterates through Match objectswhile(f.hasNext): print “found match: “ + f.next().getScore()
Pattern
Abstraction for visual patterns Are used by finding operations Methods can be chained to refine the
pattern Can define click points
Pattern(string)
Pattern(string).similar(0.9).targetOffset(10,30)
Region
Region(x, y, w, h)Region(region)Region(Rectangle)
Search in a given region Observe a region in background for changes Retrieve matches Optimize the search by chaining regions No need to be aware of the content
Act on visual objects
click( )
Actions
click(PSMLR), doubleClick(PSMLR)
dragDrop(PSMLR target, PSMLR destination)
Actions
Uses the Tesseract OCR engine
type(PSMLR, text)
text()
Actions
They can be used along with KeyModifiers
mouseDown(button), mouseUp(button)
keyDown(keys), keyUp(keys)
Actions
wait(PS), waitVanish(PS)
Actions
onAppear(PS, handler)
onVanish(PS, handler)
onChange([minChengedSize], handler)
Performance
A typical call to find() for a 100x100 target on a 1600x1200 screen takes less than 200 msec
Improve your searches
How performant do you want it to be?
Extensions
Record-playback Sikuli Guide …
Why not give it a try and make your own?
Limitations
Ok, the moment of truth!
Screenshots – unstable interfaces Visibility constraints
A paradigm shift requires a thinking shift
DemoWhat we’ve made of it