Learn how to make your drawings come alive…

COURSE:SKETCH RECOGNITIONAnalysis, implementation, and comparison of sketch recognition

algorithms, including feature-based, vision-based, geometry-based, and timing-based recognition algorithms; examination of methods to combine results from various algorithms to improve recognition using

AI techniques, such as graphical models.

Sezgin Shape Fitting:Circles and Ellipses

• Least squares fit

• Make a bounding box of stroke and form

• Oval is formed from that bounding box.

• Circle is easy: – Find out how far the circle is from the center =

d, return (d-r)^2– Ellipse more complicated

Sezgin Bezier Curve Fitting• Want to replace with a Bezier curve• http://www.math.ubc.ca/people/faculty/cass/gfx/bezier.html

– Bezier Demo• 2 endpoints and 2 control points

Sezgin Bezier Curve Fitting

• Control Point Approximation– c1 = k*t1 + v– c2 = k*t2 + u

• Notation: – Endpoints: u = p1, v = p2– T1= tangent vector – initial direction of stroke at point

p1 (u)– T2 = tangent vector of p2 (v) – initial direction of

stroke at point p2

• K = stroke length / 3– 3 is common approximation

To test curve error:

• Breaks into linear curves.

• If error to great, splits curve in half (by stroke point – not distance)

 

                                             

Bezier curve equation• http://www.cl.cam.ac.uk/Teaching/2000/AGraphHCI/SMEG/node3.ht

ml• http://www.moshplant.com/direct-or/bezier/math.html

• P0 (x0,y0) = p1, P1 = c1, P2 = c2, P3 = p2

• x(t) = axt3 + bxt2 + cxt + x0• y(t) = ayt3 + byt2 + cyt + y0• cx = 3 (x1 - x0)

bx = 3 (x2 - x1) - cxax = x3 - x0 - cx - bx

• cy = 3 (y1 - y0)by = 3 (y2 - y1) - cyay = y3 - y0 - cy - by

Matlab Curve Fitting• function [estimates, model] = fitcurvedemo(xdata, ydata)• % Call fminsearch with a random starting point.• start_point = rand(1, 2);• model = @expfun;• estimates = fminsearch(model, start_point);• % expfun accepts curve parameters as inputs, and outputs sse,• % the sum of squares error for A * exp(-lambda * xdata) - ydata, • % and the FittedCurve. FMINSEARCH only needs sse, but we want to • % plot the FittedCurve at the end.• function [sse, FittedCurve] = expfun(params)• A = params(1);• lambda = params(2);• FittedCurve = A .* exp(-lambda * xdata);• ErrorVector = FittedCurve - ydata;• sse = sum(ErrorVector .^ 2);• end• end

Yu Overview

• Recursive corner finding method– Takes stroke and tries to classify it as

• Line, Arc, Circle, Ellipse, Spiral

– If error is too great, splits stroke at highest point of curvature and tries again.

Line Recognizer

• Feature Area

Arc Recognizer

• Feature Area: Note must have best fit arc

Circle Recognition:Fit Direction Graph to Line

Circle Fit

• Use feature area to circle center

Ellipses

• Handled similarly

• Area should by symmetrical about the major and minor axes

Overtraced Circle

• Break direction graph every 2 pi• If all circles have about same radius and center point – is

circle

Spiral

• If overtraced circles have ascending or descending radii, then is a spiral (they incorrectly call this a helix in the paper)

FoundationKnowledge

IdentifyThe

Problem

ExploreInterpretations& Connections

PrioritizeAlternatives

EnvisionStrategic

Innovation

ConfusedFact

Finder

BiasedJumper

PerpetualAnalyzer

PragmaticPerformer

StrategicRe-visioner

Distinguishrelevant &irrelevant

Information

Readconflictingopinions

Relateassumptions &

biases

Analyzepros & cons

Prioritizeissues

andinformation

Justifyassumptions

Articulatevision

Reinterpretinformation

Steps for Better Thinking Performance Patterns, http://www.wolcottlynch.com

Developmental Framework for Critical Thinking

Observation Interpretation Judgment Planning

Steps in Critical

Thinking

Performance Patterns

Interventions

Step 1 Step 2 Step 3 Step 4

Computer – Sketch Recognition

• Foundation Knowledge: Stroke, Rubine Method, Sezgin Method, Yu Method

• Which primitive (or group) or gesture is this stroke?• Line, Arc, Ellipse, RGesture1, RGesture2• Precedence, Error, Tolerance• What about combination of strokes? May change lower

level interpretations… geometric context

FoundationKnowledge

IdentifyThe

Problem

ExploreInterpretations& Connections

PrioritizeAlternatives

EnvisionStrategic

Innovation

Observation Interpretation Judgment Planning

Recognizing a Line• Foundation Knowledge: Stroke, Sezgin Method, Yu Method, Geometry• How do we find *line* tolerance & error• Options:

– Least-squares error from endpoints:• + Uses endpoints• - Endpoint tails not removed• - Error may be larger than true error

– Least-square error with best fit line:• + Find best-fitting line• - Doesn’t necessarily use perceptually important start & end points• + Can remove non-perceptually important tails

– feature area:• + In theory, can be compared to other shapes• - Confusing• - Value not apparent• ? Smaller range

– ratio: euclidean length/stroke length:• + Easy to calculate• + Uses perceptually important start & end point • - Endpoint tails not removed• Doesn’t differentiate between one point being far away and several points being near• - Bigger range so harder to figure out a good threshold

– Least-squares error using best fit, but then use endpoints• - Error not same as what is chosen• + Error is more representative of line• + Perceptually important endpoints

Recognizing an Arc• Foundation Knowledge:

– Method to find sample arc as part of a circle: • Connect endpoints, find perpendicular bisector of that line• Find where that line intersects stroke• Make two lines connecting center stroke point and endpoints• Find perpendicular bisector of each line• Intersecting point is circle center• Find feature area

– A curve of order 2• Options:

– Least-squares error from endpoints with a curve of order 2:• + Uses endpoints• + Easy to compute• - Not actually an arc

– Feature area• + Uses real arc• + Faster ?• - Need the line of the arc, because takes the feature area • - Difficult – polygons could be above or below• Idea: Add threshold to radius – for comparing against line

– Least squared error with arc itself• + Uses real arc• - Harder to compute• + Faster?• Compute distance of each point to the center – subtract from radius

Recognizing a Circle• Foundation Knowledge:

– Method to find direction graph slope• Direction graph: Find direction of each point

– Direction vs. time since start» Depends on time since start

– Direction vs. point number» Depends on sampling rate

– Direction vs. stroke length» More time computationally

• Find slope• Fit a line to direction graph – use same least square method

– Splitting: Spilt it when change in direction (every) 2pi– Circle center:

• center of bounding box or• average of all points

– Circle radius:• Bounding box / 2• Average distance from center

• Options: – Slope of the direction graph == 2pi/n

• Doesn’t handle tails• Overtracing difficult because have to split

– Direction graph is linear– Circle least squares– Circle feature area

Recognizing an Ellipse• Foundation Knowledge:

– Method to find direction graph slope• Use endpoints• Find best fit line of direction graph

– Major axis and minor axis not equal– To find major axis

• Two points w/ greatest distance is the major axis– Perpendicular bisector is minor axis (where it intersects stroke)– Should points also intersect a calculated center point?

• Fit a line to the ellipse– To find center point

• Center of bounding box• Center of longest line• Center of mass

– Area of Ellipse• PI * (length of major axis/2) * (length of minor axis/2)

– Definition of Ellipse• Sum of the distance from focus 1 and focus 2 is constant• X^2 / a^2 + y^2 / b^2 = 1

– A = ½ major axis, b = ½ minor axis• Focal point is the point on the major axis that is distance ‘A’ from where minor axis intersects ellipse

• Options: – Slope of the direction graph ~2pi/n– Ellipse least squares – need foci– Ellipse feature area

• Small triangles to center vs actual ellipse area

Recognizing an Ellipse, Part 2

• Length of Major Axis– Fit a line– Longest distance

• Length of Minor Axis– Rotate Ellipse to find height of BB– Average distance of stroke points that intersect minor axis from

the major axis– Calculate from perimeter formula (p = strokelength = pi *

sqrt(2*(s^2 + b^2) – (a-b)^2/2)– Calculate distance from every point to the major axis:

• Minor axis = average distance * pi / 2

– Eigenvector method– Closest point to center

Helix Recognition

• Find major, minor axis (rotated b b)

• Find number of rotations (direction graph from circle)

• Combine n helix components

• Rotate, scale and translate

• X = cos(t) + change in x at t

• Y = sin(t) + change in y at t

Kim and Kim

• 8 possible directions in pixel space

• Resample Data of distance 5 pixels

• Compute best fit lines of 10 pixels long.

• Sharp angles smoothed out.

Curvature: Local Convextity

• When computing curvature – if curvature sign is the same (moving in same curved direction) add that to total curvature.

                                                  

Fig. 5. Algorithm 1: Curvature estimation only with local convexity

Area of Examination

• Area continues as long as it is convex.

                                                                              

Fig. 6. The effect of the adaptively changing support with k=1. The actual support is denoted by the dotted rectangle. Without convexity criterion, however, actual support is set to the solid-line rectangle.

Local Monotonicity• Check if the current direction is < min

direction – to find when it changes its sharpness

                                                       

Fig. 8. Algorithm 2: estimate curvature    at point pi.

Perceptual Recognition

• Follows the line – smoothing points in order perceptual importance.

Multipass

• Passes through the algorithm 5 times

• Each time the strictness threshold gets looser.

• Strictness = iteration number (1-5)

Examine each line segment

• Each line segment to see if want to join with line on either side.

• Line segments are sorted by length, so largest lines are processed first – Idea – want to join with a small

Threshold setting

• Messythreshold = .05 * strictness– Used to merge lines together

• Ratio = min(length1, length2) of line lengths

Research Projects

• Sketch-based Interfaces and Modeling: SBIM’08:

• Annecy, France.

• Deadline in February, 2008

• Free trip for any accepted papers

Project Idea: Cusp/Corner Detection in Complex Shapes

• Develop and implement a new corner detection algorithm that has improved recognition accuracy for complex fits. It must be fast and have greater accuracy than current methods.

Multi-stroke primitive recognition

• Extend the low level recognizers to be multi-stroke recognizers. Detail how things change. – How are they combined? How do you know when strokes should be combined?

Compare Drawing Surfaces

• Perform a user study on several drawing surfaces. See how the primitives are drawn differently on the different surfaces, and recognize them appropriately. Write up the differences and how one could shift

Project Suggestions

• Build a finite state machine recognizer for the computability class to easily draw and hand in their diagrams.

• Build a physics drawing program that attaches to a design simulator (we have interactive physics 2005)

• Build a fashion drawing program. You draw clothes on a person, and it puts them one the person.

Project Ideas

• Build a robot drawing and simulation program. You draw the robot and have a number of gestures to have it do different things

• Gesture Tetris

Project Suggestions

• Use both rubine and geometrical methods in recognition

• Develop new ways for editing.

• Build new low level recognizers

Projects!

• 2 types:– Cool application

• Sketch front end to your own research system• Fun application to go on smart board/vis contest

– Gesture Tetris– TAMU gesture-based map/directory info– Computability/Physics/EE/MechEng simulator

– New recognition algorithm• Significant change to old techniques to make a

new application

Final Project Handin• Implementation… Build it… • In class Demonstration (5-10 minutes)• Previous work

– Find at least 3 relevant papers (not read inc class)– Assign one for class to read, you lead short discussion

• Test– Run your recognition system on data.– Find out what data you need (e.g., UML class diagrams)– Each student in class will supply others data– + Find 6 more people outside (to give 15 different people)

• Paper– Introduction (why important)– Previous Work– Implementation– Results– Conclusion

Improvement Ideas

• Instead of testing the error, we test the improvement, if it is great enough, we add the vertex