CS703: PROJECT GUIDELINES

1

2

Logistics: Project• Most important part of the course

• Teams of 1 or 2 people• Expectations commensurate with size of team

• Deliverables• Oct 2: Email me a list of 3 project ideas• Oct 16: Project proposal• Nov 4: Description of progress 1• Nov 19: Description of progress 2• Dec 7-14: Presentation to the class• Dec 15: Final write-up

3

How do I choose my project?• I really want you to work on something you like!

• Ideally something that intersects with your research interests• Talk to your advisor and explain what the class is about• Ask for suggestions: “Is there a verification or synthesis problem related to

our research?”

• If you need suggestions on papers to read based on a topic ask me

• … if you really cannot find anything I can help you and guide you towards some projects

4

Verification projects• You’ll have a portfolio of program verification techniques • plenty of open-source and freely available verification tools (NuSMV, Z3,

Boogie, Uppaal)

• If you also took CS704, you can combine all you have learned

• Possible domains • Networks, GPU code, hardware binary, XML queries, CSS, GUIs, … • Think outside the box…• Example1: automatically detect whether an Android app can ever render 2 images

on top of each-other.• Example2: is CSS file 1 equivalent to CSS file 2. Which one is faster?

• It doesn’t have to use what we used in class verbatim

5

Theory projects• Least recommended (might be hard to get deliverables)

• We are seeing many models, logics, and techniques in this course• Many have open-problems, some of which are (might be) approachable

• Example: an algorithm for learning symbolic automata over the theory of BDDs• Example: prove complexity lower-bound for some problem related to automata/verification (if you want to work on this type of problem, come and talk to me)• Example: analyze complexity of synthesis questions• Example: combine models together and analyze algorithms (e.g. symbolic MSO)

AN OVERVIEW OF PROGRAM SYNTHESISEnough info so that you can think about a project

6

7

What is program synthesis?

User intent

Domain knowledge

Program can only use:Length(x), if(x)then y else z, x[i],…

Synthesizer

Input Output

Program1 -> 0

340 -> 300568 -> 500

Function f(x){If(length(x)<3)

return 0Else

return x[0]+`00’

}

For every synthesis problem…• How do you tell the system what you want?• What is the specification formalism• What is the interaction model• What happens when there is ambiguity

• How do you represent domain knowledge?• How do you guide the system towards relevant programs• How can you take advantage of the structure of the

space of programs you care about• How does the system find the program you actually want?• And how does it know it is the program you want

Different types of synthesis• Inductive Synthesis

• Functional Synthesis

• Reactive Synthesis

• Quantitative Synthesis

Traditional Machine Learning

Learn a function from a set of examples

Scalability is very important, algorithms must scale to millions of data points

Data is assumed to be noisy; need to avoid overfitting

Space of possible functions is highly stylized

Background knowledge incorporated as preprocessing and feature selection

Inductive Synthesis

Learn a function from a set of examples Scalability is not so important, usually we

are dealing with small numbers of examples

Data is assumed to be clean It’s annoying when user says f(x)=y and the

system assumes the user is wrong and decides that f(x)=z

Space of possible functions can be arbitrary

Background knowledge encoded in the description of the space and in the search itself

Inductive SynthesisSynthesize a program whose behavior satisfies a set of examples

Programming by Example: Motivation

• Two major criticisms of synthesis:• It’s too hard to make it work• Even if it works, it ends up being too hard to use

Students and Teachers

Algorithm Designers

Software Developers

End-UsersMost Useful Target

(logics, automata, etc.)

(Examples!)

FlashFill: a feature of Excel 2013 (Sumit Gulwani et al.)

Different types of synthesis• Inductive Synthesis

• Functional Synthesis

• Reactive Synthesis

Functional Synthesis

• Goal: Synthesize a function that satisfies a specification• How do we know the specification has been satisfied? Isn’t verification itself

already quite hard?• What is the relevant space of functions?• How do we explore this space efficiently?

Example:Least Significant Zero Bit

• 0010 0101 0000 0010

• Trick: • Adding 1 to a string of ones turns the next zero to a 1• i.e. 000111 + 1 = 001000

int W = 32;

bit[W] isolate0 (bit[W] x) { // W: word sizebit[W] ret = 0;for (int i = 0; i < W; i++)

if (!x[i]) { ret[i] = 1; return ret; } }

Space of possible implementations/** * Generate the set of all bit-vector expressions * involving +, &, xor and bitwise negation (~). * the bnd param limits the size of the generated expression. */

generator bit[W] gen(bit[W] x, int bnd){ assert bnd > 0; if(??) return x; if(??) return ??; if(??) return ~gen(x, bnd-1); if(??){ return {| gen(x, bnd-1) (+ | & | ^) gen(x, bnd-1) |}; }}

Example: Least Significant Zero Bit

generator bit[W] gen(bit[W] x, int bnd){ assert bnd > 0; if(??) return x; if(??) return ??; if(??) return ~gen(x, bnd-1); if(??){ return {| gen(x, bnd-1) (+ | & | ^) gen(x, bnd-1) |}; }}

bit[W] isolate0sk (bit[W] x) implements isolate0 { return gen(x, 3);}

Different types of synthesis• Inductive Synthesis

• Functional Synthesis

• Reactive Synthesis

while(true){ read inputs; make decisions; update state; write outputs;}

Reactive Synthesis

Environment System

Inputs

Outputs

Reactive Synthesis• Challenge• How do we specify correct behavior? • We now have to relate infinite sequences of inputs to infinite sequences of outputs.

• What if there is feedback between the system and environment• Often the goal of the system is to control the environment

• Very nice theory around finite state systems

Game based formalisms for Reactive Synthesis

• For every move of the adversary (ever action of the environment) the synthesized program must make a counter-move that keeps the system working correctly.• The game can be modeled with an automaton

Different types of synthesis• Inductive Synthesis

• Functional Synthesis

• Reactive Synthesis

• Quantitative Synthesis

Quantitative Synthesis

• Boolean specifications are often insufficient• They can be too hard to write• Fail to capture important properties of the problem

• What do we want to optimize?• A quantity that is inherent to the program• e.g. lines of code

• A quantity that depends on the input• e.g. running time of the program• In the worst case? The best case?• On average? On expected value? For what distribution?

24

WHAT DO THEY ALL HAVE IN COMMON

For every synthesis problem…• How do you tell the system what you want?• What is the specification formalism• What is the interaction model• What happens when there is ambiguity

• How do you represent domain knowledge?• How do you guide the system towards relevant programs• How can you take advantage of the structure of the

space of programs you care about• How does the system find the program you actually want?• And how does it know it is the program you want

26

How do you tell the system what you want?

• (Input, output) examples• Sketched program• LTL formula for reactive system• One of the above + optimization criteria

27

How do you represent domain knowledge?

• A grammar of possible operations (you can only use substring, …)• Generator: A program that can generate programs (see Sketch)• Every reactive controller will be a game automaton

28

How does the system find the program you actually want?• Enumeration-based search• Requires quick way to discard wrong solutions• Requires pruning techniques (don’t enumerate redundant functions)

• Constraint-based search• Encode the search problem into a constraint in a logic• Solve the constraint• For example Sketch encodes everything in a constraint over the integers

• Stochastic search• Randomly pick next program to search based on heuristic • Requires good heuristics

• Nice algorithms for the automata case in reactive synthesis

29

EXAMPLE APPLICATIONS

30

Inductive synthesis• Spreadsheet transformations in excel• They are typically small programs but tricky to write

• Parsers• Very natural way to provide examples (highlight part of text you want to

parse with some nonterminal)• A lot of domain knowledge to exploit

• Small recursive functional programs• Examples guide the search (each suffix is an example itself)

31

Functional synthesis• Superoptimizers (via Sketch)• They are small programs• Operate over (large) finite domains

• Query extractions (via Sketch)• Extract SQL query from Java code• Very specific theory for relations

• Program repair and automatic grading• Small edit should turn the program into a correct one• Wrong program provides a hopefully close to correct solution

32

Reactive synthesis• A theory by itself…• Can only be used by domain experts• But result is guaranteed to be correct

• BUS controllers in hardware