Automated Puzzle Generation

Simon Colton

Universities of Edinburgh and York

Background

• Train journey with Jeremy Gow– To meet Herb Simon– Puzzle generation rather than problem solving– Wrote some puzzles for Jeremy– Jeremy kept getting the “wrong” answer– Puzzle generation is a difficult task

• Reviewer’s comment– View puzzles independently of implementation

Some Example Puzzles

• Which is the odd one out?– Hair, triangles, squares, plants, words, trees– Answer: triangles (others have roots)

• Jingle is to corporation as ? Is to politician– Campaign, platform, slogan, promises– Answer: slogan

• What is next in the sequence– 4, 3, 6, 6, 2, 9 ?– Answer: later

Overview of What’s Needed

• Structure for puzzles– Characterisation of puzzles

• Puzzles must have single solutions– Theory formation helps here

• Puzzles must be of correct difficulty– Methods for disguising the answer

Queendom.com Examples

• What’s the odd one out?– Coconuts, oysters, clams, eggs, walnuts, haddock

– A: haddock (the others have shells)

• Hair is to stubble as potatoes are to ?– F.fries, sweet potatoes, potato skins, vegetable

– A. French fries

• What’s next in the sequence– 3, 8, 15, 24, 35?

– A: 48 (square integers and subtract 1)

A Characterisation of Puzzles

• Three (of many) types of puzzle are:– Odd one out, analogy, next in sequence

• Have (almost) the same structure:– Question statement – Set of choices, one of which is answer– Solution which is an embedded concept

• Some tweaking necessary to make a fit– Next in sequence puzzles have no choices– Analogy puzzles have no solution concept

Solutions to Puzzles

• Solution is a single embedded concept– Fairly simple and positively stated

• Which is the odd one out: 4, 9, 8, 36?– A: 9 (even numbers), A: 8 (square numbers)– Puzzle is unsatisfying if there are two answers

• Which is the odd one out: 2, 3, 9, 20?– A: 9 (it is a square number)

• Which is the odd one out: 23, 25, 27, 29?– A: 27 (others are primes or squares)

The Difficulty of Puzzles

• Embedded concept is usually not complex– Probably in order to ensure single solution

• Number of possible answers– Increases the search space for answer

– Could make the problem easier

• Disguising concepts– Odd one out: haddock puzzle, they’re all foodstuffs

– Next in sequence (from queendom): 2, 7, 4, 14, 6?

– Another concept interleaved (or stuck on)

The HR Program

• Automated theory formation– Concepts (ex. & def.), conjectures, proofs

– Theory is a collection of concepts (in this case)

• Concept formation via 8 production rules– Builds new concepts from old ones

– Compose,disjunct,exists,forall,match,negate,size,split

• Complexity of a concept:– Number of production rule steps

• Specialisation concepts important – Specialistion of objects of interest (e.g., prime nums)

Extension for Puzzles (General)

• HR generates theory, then builds puzzles– Embed each concept, make all puzzles, choose rep.

• From characterisation of solution:– Don’t use negate or disjunct production rules in ATF

• From single solution:– Exhaust theory up to a complexity limit

– Check for alternative solutions and discard

• From difficulty consideration– Present puzzles in order of conc. complexity, disguise

– Actively add disguise where possible

Extension for Puzzles (Special)

• User: chooses the number of possible answers (n)– Answers are presented in random order

• Odd one out:– Choose n positive and 1 negative example of spec. conc– Check all other concepts for a different solution

• Next in sequence (only in domain of integers)– Embed number type (e.g. primes, 2, 3, 5, 7, ?)– Embed function (e.g. number of divisors, 1, 2, 2, 3, ?)– Actively disguise by interleaving simple seq.

• Analogy: A is to B as C is to: D, E, F, G?– A, B, C and D share spec. property, E, F and G do not

Experiment 1: Animals

• Animals dataset (distributed with Progol)– 18 animals (dog, platypus, snake, eagle, etc.)– 12 properties (class, homeothermic, eggs, etc.)

• Theory formation up to comp. limit 5– Compose, exists, forall, match, size, split

• Asked for all odd one out & analogy puzzles– User specifies: 4 answers possible

Animals Results

• 31 puzzles about animals formed• Good examples

[15] Which is OOO: penguin, ostrich, cat, bat?[31] Eel is to platypus as shark is to snake,eagle,turtle,lizard?

• Bad example[27] Cat is to dog as eagle is to lizard, eel, ostrich, trout?

• Observations:– Low complexity of concepts, little disguise found– Need more examples of animals

• Conclusion: – Single solutions worked OK, but fairly easy to solve

Experiment 2: Integer Sequences

• Integers 1 to 30 provided– Addition, multiplication, digits, divisors– Compose, exists, match, size, split

• Theory formed up to complexity 4• Disguise simple concepts (comp. < 3)

– By interleaving other simple concepts

• All next in sequence puzzles asked for– User specifies: 6 terms of the sequence given

Sequences Results

• 24 next in sequence puzzles generated• Good examples:

[2] 4, 3, 6, 6, 2, 9, ? [numdiv, 27, mult. of 3][3] 21, 3, 24, 6, 27, 9, ? [mult 3, mult 3][10] 21, 22, 24, 25, 26, 28, ? [digit is a div]

• Bad examples:[20] 6, 0, 2, 0, 4, 0, ? [# even divisors of 24, …][22] 11, 12, 12, 13, 13, 14, ?

• Observations– Functions should start earlier on number line– Embedded concepts are in general too complex

Remarks about Creativity

• Setter: creative act is finding concept/examples• Solver: creative act is finding the answer/solution• Having a single solution:

– Want the solver to be P-creative, not H-creative

• Difference between answer and solution– IQ tests: interested in answer, not solution

• More will come to light after field testing– Comments very welcome

Conclusions and Future Work

• Characterisation of puzzles– Single pos. simp. solution, difficulty (disguise)

• Puzzle generation can be automated– Results not stunning, but still preliminary

• Puzzle generation needs improvement• Also needs hand crafting of input files• More answers/questions about puzzle

solver/setter creativity– After a field test of HR’s puzzles