Introduction to Artificial IntelligenceProf. Richard LathropProject: Monster Sudoku Solver

Monster Sudoku Final Report TemplateA template for your Final Report is below (page 5). Please, consult it as you read the following.

1. What did you do?For Part 1 of your Final Report, tell us what you did, by filling in the check-list. Please be exactly and honestly descriptive of exactly what you did and did not do. Please understand that your source code and binary executable are available to us. You will lose very many points if you claim to have done something that is not well supported by your source code and your binary executable.

Deliverable: Please fill in the template check-list in Part 1 with Yes, No, or Partial.

2. N=9 (9x9) Sudoku: Analysis of Best Methods CombinationFor Part 2 of your Final Report, you are asked to determine the best combination of methods for completing “hard” standard Sudoku problems with N=9 (9x9 grid). You are asked to complete the “hard” provided puzzles with various combinations of variable/value ordering and constraint propagation methods (FC, MRV, DH, LCV, ACP, MAP, or other extra credit advanced techniques) so as to determine which combination gives the fastest completion to “hard” puzzles. To complete a puzzle means to solve it, or prove that no solution exists, before time-out.

Deliverable: Please fill in the template in Part 2 with the numeric run-times from your runs.

Thereafter, please choose the best (fastest) combination of methods for use in further tests below. Keep that best combination of methods invariant, as determined here, for the rest of your report.

3. N = 9 (9x9) Sudoku: Estimate the Critical Value of “Hardest R”For the remainder of your Final Report, you will be asked to perform several analysis tasks based upon the best combination of methods you determined in Part 2 above. The results of these tasks are intended to educate you further about the behavior of Constraint Satisfaction Problems.

In the discussion below, M = number of initially filled-in cells of the problem, P = rows per block, Q = columns per block, and N = P*Q = grid edge length = number of distinct tokens.

Now, we can define a derivative parameter, a “difficulty ratio” R ;(R stands for the Ratio), asR = [number of initially filled cells] / [total number of cells] = M / N2

If R = 0.0 you will find a solution quickly, while if R = 1.0 you will fail quickly.

Somewhere in between the extremes of R=0.0 and R=1.0 there is a “hardest R” value, i.e., a value of R for which your program takes, on average, the longest time to find a solution or prove that there is no solution. Your task in Part 3 is to find the “hardest R9” for standard Sudoku; i.e., for N = 9, and P = Q = 3. When there is a subscript N next to R such as R N, the N refers to the board parameter N of the problem, e.g., R12 for 12 x 12 Sudoku, R16 for 16 x 16 Sudoku, etc.

Deliverable: Please fill in the template of Part 3 with graphs that model the template graphs provided for standard Sudoku (N = 9) from your system and your best combination of methods.

Graph 3.1 is run-time (y-axis) vs. R9 (x-axis) for the interval [0.1, 0.4] at 0.05 increments. Graph 3.2 is P(solvable) (y-axis) vs. R9 (x-axis) for the interval [0.1, 0.4] at 0.05

increments. The time-scale on y-axis can be either in second, micro-second, or else. If you round too

much on time-scale, you cannot observe desired result. Using microsecond or millisecond is preferred to second or else.

Estimate the “hardest” value of R for N=9, R9, from your best combination of methods.

Hardest R9 = ????

You do not need to consider values of R that are close to 0.0 or close to 1.0. You only need to bracket loosely the “hardest R,” then do local sampling within the bracket to refine your estimate. For statistical reliability, complete at least ten (10) random puzzles per R value.

Then sample various values of R in the vicinity of your initial estimate, generating and completing 10 or more random problems for each such value of R, until you are confident you have bracketed the “hardest R 9”. Sample within the bracketing interval to estimate which value of R produces the longest average time. Do additional time trials in the vicinity of that value to get a more accurate value. Use your timing data to produce the graphs in your report below.

4. What is the “Largest N” You Can Complete at the “Hardest R”?For Part 4 of your Final Report, you will determine what is the “largest N” that your system can complete reliably at the scaled “hardest R9” you determined in Part 3.

To do so, you will (1) step N through the “reasonable” values below, (2) scale M to match each new N based on your “hardest R9” from Part 3, (3) generate 10 random puzzles using your scaled “hardest” M, and then (4) find the highest N for which your system can complete 10 random problems in five minutes or less (or whatever time bound you choose). Please average at least 10 random puzzles per N value for statistical reliability. Scale M as:

Scaled M = round(N2 × R9).

For N, “reasonable” values are those that yield P and Q near sqrt(N), as given below: N = 12, P = 3, Q = 4 ;(12^2 = 144: M = 144 ×R9) N = 15, P = 3, Q = 5 ;(15^2 = 225: M = 225 ×R9) N = 16, P = 4, Q = 4 ;(16^2 = 256: M = 256 ×R9) N = 18, P = 3, Q = 6 ;(18^2 = 324: M = 324 ×R9) N = 20, P = 4, Q = 5 ; etc. N = 21, P = 3, Q = 7

N = 24, P = 4, Q = 6 N = 27, P = 3, Q = 9 N = 28, P = 4, Q = 7 N = 30, P = 5, Q = 6 N = 32, P = 4, Q = 8 N = 35, P = 5, Q = 7

For example, if N=12 and R9 =0.1, then M = round(144×0.1) = round(14.4) = 14.

Deliverable: For each of the values of N, P, and Q above, list the results for which your system can complete all 10 “hardest R” random puzzles in five minutes or less (or whatever time bound you choose). Fill in the template in Part 4 with the average run-time (total) of your system on all 10 “hardest R” random puzzles, or how many puzzles your system completed within the time bound and the average total run-time of completed puzzles.

5. Monster Sudoku: Is “Hardest R” constant as you scale up?For Part 5 of your Final Report, you will determine whether or not the N=9 “hardest R” value that you determined in Part 3 (above) is the same as the “hardest R” value for your “largest N” in Part 4. Namely, you will check if Rlargest_N is close to R9 or Rlargest_N is very different from R9.

To do so, you will repeat your analysis of Part 3, but substituting your “largest N” (and P and Q) found in Part 4. Deliverable: Please fill in the template of Part 5 with graphs that model the template graphs provided for Monster Sudoku (N = “largest N” in Part 4) from your best combination of methods.

Graph 5.1 is run-time (y-axis) vs. R (x-axis) for the interval [0.1, 0.4] at 0.05 increments. Graph 5.2 is P(solvable) (y-axis) vs. R (x-axis) for the interval [0.1, 0.4] at 0.05

increments.Finally, estimate the “hardest” value of RN from your best combination of methods.

Hardest Rlargest_N = ????

Note that you do not need to consider values of R that are close to 0.0 or close to 1.0. Please see other related remarks in Part 3, above.

Research Question: Is hardest Rlargest_N hardest R9?

Your ultimate goal is to answer the question: Does the value of “hardest R” change as N scales upward? or does the value of “hardest R” remain the same as N scales? I do not know the answer to this question, and I look forward to your results in order to answer it.

6. Turn in only one submission for each team.

If you are turnining in your team’s submission:Turn in a single “zipped” file named “yourLastName_yourUCINumericID_yourTeamName.zip” that has three directories (all items are required):

src/ ; All of your source code.

bin/ ; Your executable (must run on openlab.ics.uci.edu).doc/ ; Your documentation (this Report, see below for instructions).]

If your partner is turnining in your team’s submission:Turn in a single text file named “yourLastName_yourUCINumericID_yourTeamName.txt” that has three lines plus an affirmation (delimited by newlines):

My name, ID#, UCInetID: <Mary Roe, 99999999, mroe@uci.edu>Partner name, ID#, UCInetID (or “none”): <John Doe, 88888888, jdoe@uci.edu (or “none”)>Team name: <Keep_Calm_and_Zot_On>

7. Report Template (Required: Turn This In As Your Report)

[Anything in boldface below is required; anything below in angle brackets <> is a parameter; and anything below in square brackets [] is an instruction. The required file format for document transmission is always PDF . Students might write the report in Word or some equivalent, but thereafter they must convert their final document version to PDF for submission.]

[Turn in only ONE REPORT PER TEAM --- not one per person. Please follow the instructions on the class website about naming and file formats. Nonconforming submissions will lose points.]

Preamble: You, and How to Run Your Code[Use no more than 1 page for this section.]

My name, ID#, UCInetID: <Mary Roe, 99999999, mroe@uci.edu>Partner name, ID#, UCInetID (or “none”): <John Doe, 88888888, jdoe@uci.edu> or “none”By turning in this assignment, I/We do affirm that we did not copy any code, text, or data except CS-171 course material provided by the textbook, class website, or Teaching Staff.

The programming language(s) and versions you used in your project:[Please, describe very clearly. Java, Python, or C++; which version.]The environment needed to compile and run your project:[Please, describe very clearly; write “N/A” if not applicable.Your code MUST run and execute on openlab.ics.uci.edu]

A small write up of your implementation.[Briefly describe your Design/Implementation/Classes, etc. Code comments should appear within your code, and are not needed in your Report. Comment your code for readability.]

Part 1: (Required) What You or Your Team Did[Fill in the check box below to indicate what you or your team did. Put exactly one “X” in each triplet of Yes/Partly/No for each line item. Each Yes/Partly/No triplet must have exactly one “X”.

Please note: You will lose many points if you claim to have done something the Teaching Staff cannot reproduce. Please check exactly one of Yes/Partly/No for each item and condition below.

Please note: If you answered “I/We tested it thoroughly” as not “Yes” then you *must* answer “It ran reliably and correctly” as “No” because “Not tested thoroughly” “Not reliable.”You will lose points if you say it ran reliably and correctly but you did not test it thoroughly.]

I/We coded it. I/We tested itthoroughly.

It ran reliablyand correctly.

What was it?

Yes

Partly No Yes Partly No Yes Partly No

Required Coding ProjectBacktracking Search (BT)

Forward Checking (FC)

Minimum Remaining Values (MRV)

Degree Heuristic (DH)

Least Constraining Value (LCV)

Extra CreditWriting Your Own Shell

Writing Your Own Random Problem GeneratorArc Consistency AC-3/ACP/MAC

Local Search using Min-Conflicts HeuristicAdvanced Techniques, Extra Effort, or Creativity Not Reflected Above (*)

(*) Advanced Techniques, Extra Effort, or Creativity Not Reflected Above:[If you/your team implemented any advanced technicques, made any extra effort, or exhibited any extra creativity, that is not reflected above, please describe it briefly but clearly here.]

Part 2: N=9 (9x9) Sudoku: Analysis of Best Methods Combination

Fill in the following table based on your system’s performance on the “hard” 9x9 problems: PH1, ..., PH5 on the supplied coding shell. For each row in the table, run your system on PH1-5 with the indicated option tokens turned on, then report the average number of nodes expanded, the average time taken, and the standard deviation of the time taken. The blank line at the top is for the case of using backtracking search only; For breaking ties, order the variables lexicographically, and order the values 1 to 9. For the next seven lines, each with only one option token, run your system with only that single token enabled. (ACP, MAP, and Other are optional; if you did not do them, leave them blank.) On the next line, run FC, MRV, DH, and LCV all together. Finally, do experiments with other combinations of methods, and fill in the last blank line to indicate the combination you found to be fastest by putting an “X” under any option token that it uses.

FC MRV DH LCV (ACP) (MAP) (OTHER) AVERAGE

# NODESAVERAGE

TIME

STD. DEV.TIME

The blank row below is for the case of no heuristics and no constraint propagation.

XX

XX

(X)(X)

(X)X X X X

Fill in the blank row above with the combination you found to be fastest on PH1-5.

Did you get the results you expected? Why or why not?

Did you implement any Advanced Techniques, Extra Effort, or Creativity not reflected above? If so, please tell us what you did.

Part 3. N = 9 Sudoku: Estimate the Critical Value “Hardest R”

3.1. Fill in the following table, for total time, where R = M / N2.[Average together at least 10 different puzzles for each data point. Feel free to use different data points if it yields a more informative picture of your system running its best combination above.]

M N P Q R =M / N^2 AVERAGE

# NODESAVERAGE

TIME

STD. DEV.TIME

# (%)SOLVABLE

4 9 3 3 0.04948 9 3 3 0.098812 9 3 3 0.14816 9 3 3 0.19817 9 3 3 0.21018 9 3 3 0.22219 9 3 3 0.23520 9 3 3 0.24721 9 3 3 0.25922 9 3 3 0.27224 9 3 3 0.29628 9 3 3 0.34632 9 3 3 0.39536 9 3 3 0.444

3.2. Find the critical value of the “hardest R” for N = 9 and your best combination above.[Based on your data in 3.1 above, produce a graph similar to that shown below.]

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.400

1000200030004000

Avg Time vs. R

Avg Time

Based upon your results above, you estimate the “hardest R9” =

3.3. How does puzzle solvability for your best combination vary with R = M / N2?[Based on your data in 3.1 above, produce a graph similar to that shown below.]

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.500.000.200.400.600.801.00

Success Rate vs. R

Success R...

[For you, the x-axis will be R = M / N2, and the y-axis will be the probability that a randomly generated and completed Sudoku puzzles is solvable using your best combination above.]

3.3 Is the critical value for “hardest R” approximately the same as the value of R for which a random puzzle is solvable with probability 0.5?

4. What is the “Largest N” You Can Complete at the “Hardest R9”?

Fill in the following table using your best combination of methods from Part 2 above. Fix a time limit of 5 minutes or less to complete each new puzzle. For each new value of N, scale M with the R9 you found in Part 3 as:

Scaled M = round (N2 × R9) Consider at least 10 random puzzles for each new value of N. Report the number and percentage of random puzzles that your best combination was able to complete for that N in 5 minutes or less. For those puzzles that your system completed, report average nodes, average runtime, and standard deviation of the runtime. (Standard deviation is “none” if your system completed only one puzzle.) When your system fails to complete any puzzles for some value of N, it is unnecessary to continue.

“Hardest M”round (N2 × R9)

N P Q

# (%) Completed

in 5 Minutes or Less

AVERAGE# NODES

(Completed puzzles only)

AVERAGETIME

(Completed puzzles only)

STD. DEV.TIME

(Completed puzzles only)

12 3 415 3 516 4 418 3 620 4 521 3 724 4 627 3 928 4 730 5 632 4 835 5 7

Part 5. Monster Sudoku: Is “Hardest R” constant as you scale up?

5.1. Fill in the following table. [Fill in N, P, and Q from the largest value of N in Part 4 for which you completed all puzzles in 5 minutes or less. Then, try different M values that are computed from R values tried in Part 3. For example, new M values can be round (0.0494 * N_largest2), round (0.0988 * N_largest2), etc. Note that 0.0494, 0.0988, are came from the first two rows in table in Part 3. The actual values of M that you use will result in slightly different values of R because of rounding, so recalculate R below for the values of M you actually used. Average together at least 10 different puzzles for each data point. Feel free to use different data points if it yields a more informative picture of your system running its best combination above.]

M N P Q R =M / N2

AVERAGE# NODES

AVERAGETIME

STD. DEV.TIME

# (%)SOLVABLE

5.2. Find the critical value of the “hardest R” for N_largest.[Based on your data in 5.1 above, produce a graph similar to that shown below.]

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.400

1000200030004000

Avg Time vs. R

Avg Time

Based upon your results above, you estimate the “hardest RN_Largest” =

5.3. How does puzzle solvability for your best combination vary with R = M / N2?[Based on your data in 5.1 above, produce a graph similar to that shown below.]

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.500.000.200.400.600.801.00

Success Rate vs. R

Success R...

[For you, the x-axis will be R = M / N2, and the y-axis will be the probability that a randomly generated standard Sudoku puzzles is solvable at all using your best combination above.]

5.4 Is the critical value for “hardest R” approximately the same as the value of R for which a random puzzle is solvable with probability 0.5?

5.5 Research Question: Is hardest_Rlargest_N hardest_R9? Why or why not? How is the hardest R related to the board parameter N? What is the shape of the function R(N), which gives you the hardest ratio for 9, 12, 15, 16, 18, 20, …, 35?

Appendix: Extra Credit: (*) Advanced Techniques, Extra Effort, or Creativity Not Reflected Above:

[If you/your team implemented any advanced techniques, made any extra effort, or exhibited any extra creativity, then you described it briefly but clearly above in Part 2. Here, please place any data that documents what you say you did.]