Roles of Data Mining and Machine Learning in Computational ......obtained using machine learning methods: A. A predictive (or discriminative) model can label given instances – E.g

Roles of Data Mining and Machine Learning

in Computational Creativity PROSECCO Tutorial about Computational Creativity

ICCC 2016, Paris

Hannu Toivonen, University of Helsinki, Finland [email protected]

Based on joint work with Oskar Gross 27/06/16 1

www.helsinki.fi/yliopisto

–  A purely preprogrammed generative system –  only does what it was told to do –  has little if any creativity

–  Adaptivity or self-determinism –  is necessary to attribute any originality, responsibility or

creative autonomy to a creative system –  Transformative or meta-level creativity (cf. Boden,

Wiggins) can be attributed with higher creativity –  …but how to build a system to deal with unanticipated

cases? → Opportunities for ML and DM

27/06/16 2

The opportunity for Data Mining (DM) and Machine Learning (ML)


–  To highlight some possible ways of using DM and ML in computational creativity –  (This is not a tutorial on DM and ML methods)

–  A limited number of examples will be given –  More information, examples and references can be

found in our review paper: Hannu Toivonen and Oskar Gross: Data Mining and Machine Learning in Computational Creativity. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 5 (6): 265-275. 2015. (link)

27/06/16 3

Goals of this talk


– Covered already by Wiggins in the previous parts of the tutorial –  What is creativity, computational creativity –  Creativity as search, transformational creativity

–  As a recap, consider the definition of creative autonomy by Jennings (2010)

27/06/16 4

Background on CC


1.  Autonomous Evaluation The system can evaluate its liking of a creation without seeking opinions from an outside source. -  Any opinion is formed by the system itself -  However, it may consult others at other times -  Examples: preprogrammed evaluation, evaluation

function learned from the user

Criteria for Creative Autonomy (1/3), Jennings (2010)


2.  Autonomous Change The system initiates and guides changes to its standards without being explicitly directed when and how to do so. -  External events and evaluations may prompt and guide

changes -  The system decides when and how to change them -  The system decides if new standards are acceptable -  Fixed or learned evaluation functions can be used to

bootstrap the process

Criteria for Creative Autonomy (2/3)


3.  Non-Randomness The system’s evaluations and standard changes are not purely random. -  The two first criteria could be easily met by random

decisions -  Not all randomness is excluded, however

Criteria for Creative Autonomy (3/3)


–  Let’s use a simple generate-and-test model to illustrate uses of machine learning and data mining in CC (computational creativity)

–  Assessing one’s own output is an elementary creative responsibility (and Jennings’ 1st criterion)

27/06/16 8

Illustrative setup: The generate-and-test model

gen() eval(a) Artefact a Ok?

no

yes

Learning to evaluate

27/06/16 9

gen() eval(a) Artefact a Ok?

no

yes


– Use ML to learn an evaluation function eval(a) from training examples –  Supervised learning (from given examples) –  E.g. a classifier that tells if the result is good

–  Assuming a generator gen() exists, its outputs can now be filtered by the trained classifier without explicit directions by the programmer –  More responsibility over the evaluation step –  Fulfills Jenning’s first criteria of creative autonomy

27/06/16 10



An example system: DARCI (Ventura et al) – Creates images that express an emotion –  Emotion detection is based on artificial neural

networks trained by users of the system –  A genetic algorithm is used as generator gen()

–  Adapts to the evaluation/fitness function eval()

–  http://darci.cs.byu.edu/ –  ”DARCI, draw me a happy picture!”

27/06/16 11


www.helsinki.fi/yliopisto A happy image by DARCI, http://darci.cs.byu.edu/ 27/06/16 12


Bottlenecks in learning the eval() function –  Learning an evaluation (or fitness) function eval(a)

can be very difficult –  E.g., how does one evaluate the quality of a poem?

– Generating complex artefacts, i.e., producing the function gen(), can be very hard –  In practice, the generation step must be adaptive in

order to be effective

– Can lead to pastiche generation, i.e., mere imitation of training examples rather than creativity

27/06/16 13


Learning to generate

27/06/16 14

gen() test(a) Artefact a Ok?

no

yes


Roughly speaking, two kinds of models can be obtained using machine learning methods: A.  A predictive (or discriminative) model can label

given instances –  E.g. a neural network or a support vector machine can

categorize a given example to one of the trained categories

B.  A generative model can produce new instances –  E.g. a Bayesian model for the joint distribution of all

values can be used to generate new instances from the joint distribution

27/06/16 15

Learning to generate


1.  Completion of partial artefacts

–  Given some part of the artefact, predict the values of the remaining parts

–  Can be based on training on complete artefacts Example: harmonization of music: –  Given a melody (possibly created by the system itself),

choose suitable chords to accompany the melody

27/06/16 16

A. Learning to generate using predictive models


2.  Reduce the task of generating complex structures to selection.

Example (revisited): Generation of accompaniment for a melody by running a classifier to pick a suitable chord as well as a suitable pattern

27/06/16 17



3.  Generate complex structures using instance-based techniques

–  E.g. k-nearest neighbours and case-based reasoning –  avoids using models, decision structures, or patterns

‒  models could be difficult to specify or learn

‒  models could be restrictive

27/06/16 18



Example: Corpus-based poetry by Toivanen et al. –  Background: typical inputs to a poetry writing system

a.  linguistic knowledge, e.g. a grammar or templates b.  world knowledge, e.g. a knowledge base

–  Poetry generation by Toivanen et al a.  instance-based generation (copying) of grammar

using a large corpus of poetry b.  obtains statistical world knowledge automatically by

mining patterns from (another) corpus

27/06/16 19



–  Example of operation of the method (translation from Finnish)

–  An original piece of text is selected (randomly): –  how she once played in a big, green park […]

–  Topic for new poem: (children’s) play – 

27/06/16 20


she then whispering played daring in a daring, how under the pale trees. She had heard for fun how her whispering drifted as jingle to the wind.


Music swells, accent practises, theatre hears! Her delighted epiphanies bent in her universe: – And then, singing directly a universe she disappears! An anthem in the judgements after verse!

27/06/16 21


Generative models can be used more directly to generate artefacts (or to complete partial ones) –  E.g. Markov models for sequencies such as text and

music –  (Bayesian) models of joint probability distributions –  Some artificial neural networks

27/06/16 22

B. Learning to generate using generative models


An example: Deep Dream by Google –  A neural model trained with existing images –  Learns their features and can generate (or

complement or re-render) images –  http://deepdreamgenerator.com

27/06/16 23

B. Learning to generate using generative models

www.helsinki.fi/yliopisto 27/06/16 24

www.helsinki.fi/yliopisto 27/06/16 25

Mining patterns for creative tasks

27/06/16 26


Basic idea: 1.  Use data mining to discover patterns in, say, text 2.  Utilize these patterns in a generation function gen() Examples: –  Corpus-based poetry of Toivanen et al

–  Words used as substitutes come from text mining

–  Metaphor generation (Veale et al)

27/06/16 27



Example: metaphor generation (Veale et al) –  Extraction step: extract similes from a corpus

–  Look for patterns of the form “as P as a/an V”, e.g. “as strong as an ox”

–  “strong” is a stereotypical property of “ox” if the pattern “as strong as an ox” occurs often

–  Metaphor construction step: use stereotypical information –  To express that ”John is stupid” in a metaphorical way, pick

a noun V for which ”stupid” is a stereotypical property –  Donkey is found as stereotypically stupid –  Compare John to donkey to express that he is stupid

http://ngrams.ucd.ie/metaphor-eye/

27/06/16 28



Metaphor-Eye Why are scientists like artists? –  Scientists

–  …develop ideas like artist –  …explore ideas like artist –  …acquire skills like artist –  …spread ideas like artist –  …nurture ideas like artist –  …develop techniques like artist

27/06/16 29

Learning in transformational creativity

27/06/16 30


– U: universe of all possible (partial) artifacts – R: rules that define the search space –  E: rules that evaluate an artifact –  TR,E: traversal rules to search for artifacts that satisfy

R and E

–  Transformational creativity takes place when R, E or T is modified by the system itself

– Next: a quick review of issues and opportunities for ML, proposed by Wiggins

27/06/16 31

Recall Wiggins’ creative search


Wiggins suggests uses of ML/DM: –  Automatic adaptation of R or T

–  To remedy aberration: use aberrant concepts as positive or negative examples, depending on their value

–  To remedy generative uninspiration: use positive (and negative) examples received from outside

–  Automatic adaptation of E –  Use feedback and evaluations received from outside

(not covered by Wiggins)

Transformational Creativity Using Data Mining and Machine Learning


–  “Generative uninspiration”: TR,E does not reach anything valued by E

–  A milder form: a lot of (highly) valued concepts are unreachable by TR,E

–  Transformation of TR,E is required – Help from outside is needed, e.g., valued (and

unvalued) concepts not reachable to TR,E –  use them as positive (and negative) training examples

to modify TR,E

Generative uninspiration


–  “Aberration”: TR,E reaches concepts outside R –  “Productive aberration”: TR,E reaches some valued

concepts outside R –  “Pointless aberration”: the concepts outside R are

not valued by E either

– Need to transform T to avoid useless search –  Possibly transform R to include the valued concepts –  Learning: use aberrant concepts as positive or

negative examples, depending on their value

Aberration

Conclusion

27/06/16 35


–  Learning to evaluate artifacts –  Learning to generate artifacts

–  possibly by reducing generation to selection, or to a sequence of predictive steps

– Mining patterns to be used in generation – Doing these adaptively also during the runtime –  Also learning/adapting search space R (if possible) –  Learning to frame (cf. FACE by Colton et al) – Higher meta-levels: Learning to produce new

generation, evaluation and search functions

27/06/16 36

Examples of creative responsibilities assumed by DM/ML


For more information and citations see the following two survey papers: –  Hannu Toivonen and Oskar Gross: Data Mining and

Machine Learning in Computational Creativity. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 5 (6): 265-275. 2015.

–  Ping Xiao, Hannu Toivonen, et al: Review of Conceptual Representations for Concept Creation. Manuscript.

27/06/16 37

Literature


–  Margaret A. Boden: Creative Mind: Myths and Mechanisms (2nd ed.). Routledge, Oxon, UK, 2004 (1st edition:1992)

–  Simon Colton and Geraint A. Wiggins: Computational Creativity: The Final Frontier? ECAI 2012 - 20th European Conference on Artificial Intelligence, 21-26, Montpellier, France, August 2012.

–  Wiggins, Geraint A : A preliminary framework for description, analysis and comparison of creative systems. Knowledge-Based Systems, 19 (7), 449—458, 2006.

–  K.E. Jennings: Developing creativity: Artificial barriers in artificial intelligence. Minds and Machines 20(4): 489-501, 2011.

Some key references

Documents

Roles of Data Mining and Machine Learning in Computational ......obtained using machine learning methods: A. A predictive (or discriminative) model can label given instances – E.g