Artificial Intelligence for the Film Industry

Artificial Intelligencefor the Film Industry

Georg RehmDFKI, Germany

Propellor FilmTech Meetup #1 – 25 July 2017 – Berlin, Germany

• Sites in Saarbrücken, Kaiserslautern, Bremen, Berlin, Osnabrück, St. Wendel

• Intelligent software systems: robotics, agents, image processing, language understanding, augemented reality, 3D, knowledge management,HMI, security, Industrie 4.0.

• 900+ staff – ca. 300 running projects• CEO: Prof. Dr. Wolfgang Wahlster

Propellor FilmTech Meetup #1 – 25 July 2017

Deutschland-GmbH

2

German Research Centre forArtificial Intelligence GmbH (founded in 1988)

Propellor FilmTech Meetup #1 – 25 July 2017 3




Artificial Intelligence• Strong AI: hypothetical machine with a consciousness

and behaviour at least as flexible as that of a human.• Weak AI: software without consciousness, tailored to

one specific purpose and task.• Machine Learning: give “computers the ability

to learn without being explicitly programmed” (Arthur Samuel, 1959)

• Examples: pattern recognition (e.g., handwriting), predictions (stock exchange), recommendations (films!) etc.



Data Intelligence

Current breakthroughs with machine learning methods(Deep Learning). Also still in use: symbolic, rule-based methods

Language Technology• Language Technology makes use of theoretical results

in linguistics in marketable solutions and applications.• Uses research results from:

– Artificial Intelligence– Computer Science– Computational Linguistics

• Natural Language Processing• Natural Language Understanding

– Psychology, Psycholinguistics– Cognitive Science

• Language: Next big thing for AI!


Example Applications• Spellchecker• Dictation systems• Translation systems• Search engines• Report generation• Expert systems• Dialogue systems• Text summarisation

AI and the Film Industry• AI and Language Technology:

Many breakthroughs in multiple different application areas

• Focus: Film industry

• Massive potential!


FilmIndustry

Language Technology

AI andDeep

Learning

Big DataFast

machines and

networks

Internet of Things

! Editing Trailers

! Writing Scripts

! Recommenders




• Simple Machine Learning• Training data: 100 trailers• Create model and apply it

(i.e., to the film “Morgan”)• Watson selected scenes• “A human editor was still

needed to patch thescenes together to tell a coherent story.”


• Good example of usingtech in a curation setting

• With the machine you’re faster but you arrive at thesame result as the human

• The “AI” part is attributedto the technology by the(astonished) human who’s also been influenced by clever marketing

• Note: an “AI” is only good at one very specific task!



• No, it didn’t.• This is fake news

(category: clickbait).




• Simple ML again• Training data: scripts

of sci-fi movies• Neural network learns

patterns and is able to generate a new script

• Deep Learning for Natural Language Generation (NLG)

• Can also be applied to Shakespeare


• Simple ML again• Training data: scripts

of sci-fi movies• Neural network learns

patterns in scripts and is able to generate new script

• Deep Learning for Natural Language Generation (NLG)

• Can also be applied to Shakespeare

To me, fair, so you never be,Each trifle, way, when bore your beauty when,Such hence your can still,O thou how much were your self the wrong chide.

Thy youth’s time and face his form shall cover?Now all fresh beauty, my love there,Will ever time to greet, forget each, like ever decease, But in a best at worship his glory die.

Stanley Xie, Ruchir Rastogi, Max Chang: “Deep Poetry: Word-Level and Character-Level Language Models for Shakespearean Sonnet Generation” (Stanford)


• The automatically generated script doesn’t make any sense whatsoever.

• “Sunspring” is an interesting exercise but, essentially, unwatchable.

AI – Taking StockWhat AI is good at• Identifying patterns• Extracting structure• Data analysis• Mimicking regularities• Important: training data

(ideally structured)• Emulating smart

behaviour

What AI is really bad at• Creativity• Eloquence• Curiosity• Freshness• Originality• Poetry• Out-of-the-box’ness• Understanding of the

world that surrounds us


The Outer Limits

AI would even fail at thisseemingly simply task …




Even “automatic mockbuster generation” required a level of creativity that is way beyond

anything Artificial Intelligence can achieve today.


https://medium.com/@bootstrappingme/the-german-artificial-intelligence-landscape-b3708b325124

Film AI Startups• VaultML, ScriptBook, Pilot Movies: Project ticket sales

and box office performance (script or trailer analysis)• Iris.tv: Better recommendations• Qloo: Cultural AI, predicts the tastes for any target

audience and maps relationships (music, books, films)• Valossa: Detects people, context, topics etc. in video

and audio streams (assist video content discovery)• Cinuru: Customer Relationship Management• Much more can be done …


http://www.nanalyze.com/2017/07/6-startups-ai-movies-entertainment/

Data for Film AI• Current AI methods can do a lot with interesting data.• What is “interesting data” in the film industry?• Could be anything from every part of the life cycle:

– Scripts – Preferences – List of scenes– Reviews – Films watched – Credits– Emails – Categories – Rankings– Production notes – Genres – Relations– Demographics – Lexicons – Databases– Statistics – Focus groups – Marketing– Box office results – Target audience – etc.


Example Use Case• Let’s have a look at a concrete use case and challenge• Deep, context-aware recommendations that fit the

viewer’s mood, time constraints, interests, focus areas• Example: you have ca. 60 minutes, you’re interested in

current politics in the US, have an upcoming trip to Vancouver, like running, AI, languages and technology

• Recommender could suggest films or documentaries that exactly fit this bill (using a deep user model)

• How? By pulling different sources of data together• Calendar (upcoming trips and meetings), browsing and

search history, to do list, social media, IMDB profile etc.


Example: Details• Data sources:

– Calendar: upcoming trip to Vancouver– To do list: prepare the trip (e.g., “find running route”)– Email archive: hotel booking in Vancouver

• The smart recommender algorithm could examine these data points and help the user get a few things done

• Upcoming trip + likes running + location of hotel = videos of running routes or running clubs in Vancouver

• Upcoming trip + likes running = films about, or including, running that are set in or that were shot in Vancouver


Lifelogging and IoT• Lifelogging = record your whole life• Mobile phones and activity trackers

are getting closer (quantified self)• Measuring heart-rate 24/7/365• Advanced measurements like

VO2 max through several sensors is consumer-grade technology!

• What about film-related data points?• Measuring excitement, boredom, attention, repetition,

amazement, imitation, cringe-worthiness, disgust, tenseness, eye-tracking etc.


https://en.wikipedia.org/wiki/Lifelog

Film and Quantified Self• Vision: create deep user models by pulling together a

user’s heterogeneous information and data streams (calendar, contacts, to do lists, profiles, youtube etc.)

• Add lifelogging-related data by tapping into activity trackers, smart watches, mobile phone sensors

• Endless possibilities would emerge … – and will!• Measure the reactions of one viewer or a whole theater

by measuring their vital stats when watching a film• Revolutionise film development and test screenings• Adapt films dynamically (insert explosion when bored)


• Propellor | Forum #1 created intriguing results

• Any Film, Anywhere – user model, watchlist, loc, reco

• Bubble Buster – user model, reco (safe & surprising)

• Super AI Brain – user model, reco

• Data of the Movie – user model, reco, biofeedback

• AI-based Storytelling – user model, audience clustering, Big Data-based storytelling


http://www.propellorfilmtech.com/forum

Challenges• Integration of heterogeneous data sources (from silos!)

into a unified and homogeneous model as well as making this model available to recommender algorithms.

• Getting the data is hard, so is mapping the data.

• How do we get – on a very large scale – the data out of connected devices (smart phones, smart watches, activity trackers, tv sets etc.) into our own applications?

• The typical, very hard, AI challenges: How can we reallymodel creativity, originality etc.?


Thank you!


DKT kick-off meeting – 25 September 2015

Digital Curation Technologies• Support and optimise digital curation through language and

knowledge technologies• Develop innovative prototypes together with the SME partners• Further develop DFKI technologies and transfer them into

industry through platform for digital curation technologies

Georg Rehm und Felix Sasaki. “Digital Curation Technologies.” In Proceedings of the 19th Annual Conference of the European Association for Machine Translation (EAMT 2016), Riga, Lettland, Mai 2016

Georg Rehm und Felix Sasaki. “Digitale Kuratierungstechnologien – Verfahren für die effiziente Verarbeitung, Erstellung und Verteilung qualitativ hochwertiger Medieninhalte.” In Proceedings der Frühjahrstagung der Gesellschaft für Sprachtechnologie und Computerlinguistik (GSCL 2015), S. 138-139, Duisburg, 2015

Sprach- und Wissenstechnologien

Kuratierungstechnologien

Branchentechnologien

Plat

tform

tech

nolo

gie

Branchenlösungen

http://digitale-kuratierung.de