Introduction to Functional Reactive Programming through ...webuser.uni-weimar.de/~xume0027/lecture02-frp.pdf · Introduction to Functional Reactive Programming through Games and Music

Introduction to FunctionalReactive Programming through

Games and MusicBauhaus-Universität Weimar, 10–13 April 2017

Henrik Nilsson

Functional Programming Laboratory, School of Computer Science

University of Nottingham, UK

Introduction to Functional Reactive Programming through Games and Music – p.1/50

FP in the Real World (1)

Many eloquent and compelling cases forfunctional programming in general:




• John Backus, 1977 ACM Turing AwardLecture: Can Programming Be Liberated fromthe von Neumann Style?





• John Hughes, recent retrospective: WhyFunctional Programming Matters(on YouTube, recommended)





• John Hughes, recent retrospective: WhyFunctional Programming Matters(on YouTube, recommended)

One key point: Program with whole values, not aword-at-a-time. (Will come back to this.)



But how can pure functions:




• Communicate with the outside world?





• React in a timely way to events?





• React in a timely way to events?

• Deal with state?

And so on.



One answer:



One answer: Forget about purity!




• Allow functions to have side effects:state, exceptions, I/O, . . .

• Some benefits of functional style retained






• Approach of languages like Lisp, Scheme,ML, OCaml, Erlang, Scala, Erlang







However:







However:

• Can make code harder to understand







However:


• Impairs formal reasoning







However:


• Impairs formal reasoning

• Very poor fit with lazy evaluationIntroduction to Functional Reactive Programming through Games and Music – p.4/50


Another answer:



Another answer: Monads!




• Allows effects in a pure setting





• Flexible: any kind of interfacing or imperativealgorithms can be expressed






• Crucial to the success of pure languages likeHaskell






• Crucial to the success of pure languages likeHaskell

However, whatever fragments of the code areexpressed in an effectful manner comes with theusual caveats: in particular, often an imperative,“word-at-a-time” programming style.



Here, we are going to consider a third option:

• Reactive systems described through purefunctions over time-varying entities.

• Whole-value approach, as opposed toword-at-a-time.



Here, we are going to consider a third option:

• Reactive systems described through purefunctions over time-varying entities.

• Whole-value approach, as opposed toword-at-a-time.

Reactive systems:

• Input arrives incrementally while system running.

• Output is generated in response to input in aninterleaved and timely fashion.

Contrast transformational systems.


Functional Reactive Programming



• Key idea: Don’t program one-time-step-at-a-time,but describe an evolving entity as whole.




• FRP originated in Conal Elliott and Paul Hudak’swork on Functional Reactive Animation (Fran).




• FRP originated in Conal Elliott and Paul Hudak’swork on Functional Reactive Animation (Fran).Highly cited 1997 ICFP paper; ICFP award formost influential paper in 2007.





• FRP has evolved in a number of directionsand into different concrete implementations.





• FRP has evolved in a number of directionsand into different concrete implementations.

• We will use Yampa: an FRP systemembedded in Haskell.


FRP Applications

Some domains where FRP or FRP-inspiredapproaches have been used:

• Graphical Animation

• Robotics

• Vision

• Sound synthesis

• GUIs

• Virtual Reality Environments

• Games


FRP Applications

Some domains where FRP or FRP-inspiredapproaches have been used:

• Graphical Animation

• Robotics

• Vision

• Sound synthesis

• GUIs

• Virtual Reality Environments

• GAMES


Games in FRP (1)

We will illustrate the FRP approach and Yampaby considering simple video games. Our wholevalues are things like:

• The totality of input from the player

• The animated graphics output

• The entire life of a game object


Games in FRP (1)

We will illustrate the FRP approach and Yampaby considering simple video games. Our wholevalues are things like:

• The totality of input from the player

• The animated graphics output

• The entire life of a game object

We construct and work with pure functions on these:

• The game: function from input to animation

• In the game: fixed point of function oncollection of game objects


Games in FRP (2)

Play Store: Pang-a-lambda (Keera Studios)


Games in FRP (3)

Lots of bouncing balls in Pang-a-lambda!

y = y0 +

∫v dt

v = v0 +

∫−9.81

On impact (fully elastic collision):

v = −v(t−)

We are used to describing behaviours in theirtotallity over time in mathematical models. Whynot program in the same way?


Key FRP Features

Combines conceptual simplicity of the synchronousdata flow approach with the flexibility and abstractionpower of higher-order functional programming:


Key FRP Features


• Synchronous


Key FRP Features


• Synchronous

• First class temporal abstractions


Key FRP Features


• Synchronous


• Hybrid: mixed continuous and discrete time


Key FRP Features


• Synchronous



• Dynamic system structure


Key FRP Features


• Synchronous



• Dynamic system structure

Good fit for typical video games, inter alia,(but not everything labelled “FRP” supports them all).


Yampa


Yampa

• FRP implementation embedded in Haskell


Yampa


• Key notions:

- Signals: time-varying values

- Signal Functions: pure functions on signals

- Switching: temporal composition of signalfunctions


Yampa


• Key notions:

- Signals: time-varying values

- Signal Functions: pure functions on signals

- Switching: temporal composition of signalfunctions

• Programming model:


Yampa?


Yampa?

Yampa is a river with long calmly flowing sectionsand abrupt whitewater transitions in between.

A good metaphor for hybrid systems!Introduction to Functional Reactive Programming through Games and Music – p.14/50

Signal Functions


Signal Functions

Intuition:


Signal Functions

Intuition:

Time ≈ R


Signal Functions

Intuition:

Time ≈ R

Signal a ≈ Time → a

x :: Signal T1y :: Signal T2


Signal Functions

Intuition:

Time ≈ R


x :: Signal T1y :: Signal T2SF a b ≈ Signal a → Signal bf :: SF T1 T2


Signal Functions

Intuition:

Time ≈ R


x :: Signal T1y :: Signal T2SF a b ≈ Signal a → Signal bf :: SF T1 T2

Additionally, causality required: output at time t

must be determined by input on interval [0, t].


Signal Functions and State

Alternative view: Signal functions canencapsulate state:

state(t) summarizes input history x(t′), t′ ∈ [0, t].


Signal Functions and State

Alternative view: Signal functions canencapsulate state:

state(t) summarizes input history x(t′), t′ ∈ [0, t].

From this perspective, signal functions are:

• state(t)

• stateless if y(t) depends only on x(t)


Some Basic Signal Functions

identity :: SF a a



identity :: SF a a

constant :: b → SF a b



identity :: SF a a


iPre :: a → SF a a



identity :: SF a a


iPre :: a → SF a a

integral :: VectorSpace a s ⇒ SF a a

y(t) =

t∫

0

x(τ) dτ


Composition

In Yampa, systems are described by combiningsignal functions (forming new signal functions).


Composition


For example, serial composition:


Composition



A combinator that captures this idea:

(≫) :: SF a b → SF b c → SF a c


Composition



A combinator that captures this idea:

(≫) :: SF a b → SF b c → SF a c

Signal functions are the primary notion; signals asecondary one, only existing indirectly.


Systems

What about larger, more complicated networks?How many combinators are needed?


Systems

What about larger, more complicated networks?How many combinators are needed?

John Hughes’s Arrow framework provides agood answer!


The Arrow framework (1)

arr f f ≫ g

first f loop f

arr :: (a → b)→ SF a b

(≫) :: SF a b → SF b c → SF a c

first :: SF a b → SF (a, c) (b, c)

loop :: SF (a, c) (b, c)→ SF a bIntroduction to Functional Reactive Programming through Games and Music – p.20/50

Arrow notation


Arrow notation


Arrow notation

proc x → do

rec

u ← f −≺ (x , v)

y ← g−≺ u

v ← h−≺ (u, x )

returnA−≺ y


Arrow notation

proc x → do

rec

u ← f −≺ (x , v)

y ← g−≺ u

v ← h−≺ (u, x )

returnA−≺ y

Only syntactic sugar:

everything translated into a

combinator expression.


A Bouncing Ball

Lots of bouncing balls in Pang-a-lambda!Consider 1D case:

y = y0 +

∫v dt

v = v0 +

∫−9.81

On impact:

v = −v(t−)

(fully elastic collision)


Modelling the Bouncing Ball: Part 1

Free-falling ball:

fallingBall :: Pos → Vel → SF () (Pos ,Vel)

fallingBall y0 v0 = proc ()→ do

v ← (v0+) <̂< integral−≺ − 9.81

y ← (y0+) <̂< integral−≺ v

returnA−≺ (y , v)



Free-falling ball:

fallingBall :: Pos → Vel → SF () (Pos ,Vel)

fallingBall y0 v0 = proc ()→ do

v ← (v0+) <̂< integral−≺ − 9.81

y ← (y0+) <̂< integral−≺ v

returnA−≺ (y , v)

Some different and extra symbols, but just superficialsyntactic details: the structure remains the same.We have turned the mathematical model into adeclarative program!


Events

Yampa models discrete-time signals by lifting theco-domain of signals using an option-type:

data Event a = NoEvent | Event a

Discrete-time signal = Signal (Eventα).


Events

Yampa models discrete-time signals by lifting theco-domain of signals using an option-type:

data Event a = NoEvent | Event a

Discrete-time signal = Signal (Eventα).

Some functions and event sources:

tag :: Event a → b → Event b

after :: Time → b → SF a (Event b)

edge :: SF Bool (Event ())



Detecting when the ball goes through the floor:

fallingBall ′ ::

Pos → Vel → SF () ((Pos ,Vel),Event (Pos ,Vel))

fallingBall ′ y0 v0 = proc ()→ do

yv@(y , )← fallingBall y0 v0−≺ ()

hit ← edge −≺ y 6 0

returnA−≺ (yv , hit ‘tag ‘ yv)


Switching

Q: How and when do signal functions “start”?


Switching


A: • Switchers apply a signal functions to itsinput signal at some point in time.


Switching



• This is temporal composition of signalfunctions.


Switching




Switchers thus allow systems with varyingstructure to be described.


Switching




Switchers thus allow systems with varyingstructure to be described.

Generalised switches allow composition ofcollections of signal functions. Can be used tomodel e.g. varying number of objects in a game.


The Basic Switch

Idea:

• Allows one signal function to be replaced byanother.

• Switching takes place on the first occurrenceof the switching event source.

switch::

SF a (b,Event c)

→ (c → SF a b)

→ SF a b


The Basic Switch

Idea:



switch:: Initial SF with event source

SF a (b,Event c)

→ (c → SF a b)

→ SF a b


The Basic Switch

Idea:



switch:: Function yielding SF to switch into

SF a (b,Event c)

→ (c → SF a b)

→ SF a b



Making the ball bounce:

bouncingBall :: Pos → SF () (Pos ,Vel)

bouncingBall y0 = bbAux y0 0.0

where

bbAux y0 v0 =

switch (fallingBall ′ y0 v0 ) $ λ(y , v)→

bbAux y (−v)


Game Objects

data Object = Object {objectName :: ObjectName

, objectKind :: ObjectKind

, objectPos :: Pos2D

, objectVel :: Vel2D

. . .

}

data ObjectKind = Ball . . . | Player . . . | . . .

data ObjectInput = ObjectInput

{userInput :: Controller

, collisions :: Collisions

}Introduction to Functional Reactive Programming through Games and Music – p.29/50

Overall Game Structure

gamePlay :: [ListSF ObjectInput Object ]

→ SF Controller ([Object ],Time)

gamePlay objs = loopPre [ ] $

proc (input , cs)→ do

let oi = ObjectInput input cs

ol ← dlSwitch objs−≺ oi

let cs ′ = detectCollisions ol

tLeft ← time−≺ ()

returnA−≺ ((ol , tLeft), cs ′)

ListSF and dlSwitch are related abstractions thatallow objects to die or spawn new ones.


And now for something different . . .

Switched-on Yampa: Programming ModularSynthesizers in Haskell


And now for something different . . .

Switched-on Yampa: Programming ModularSynthesizers in Haskell


Modular synthesizers?





Steve Pocaro, Toto,with Polyfusion Syn-thesizer


Modern Modular Synthesizers


Where does Yampa enter the picture?



• Music can be seen as a hybrid phenomenon.Thus interesting to explore a hybrid approachto programming music and musical applications.




• Yampa’s programming model is very reminiscentof programming modular synthesizers:




• Yampa’s programming model is very reminiscentof programming modular synthesizers:

• Fun application! Useful for teaching?Introduction to Functional Reactive Programming through Games and Music – p.34/50

What have we done?


What have we done?

Framework for programming modularsynthesizers in Yampa:


What have we done?


• Sound-generating and sound-shapingmodules


What have we done?



• Additional supporting infrastructure:

- Input: MIDI files (musical scores), keyboard

- Output: audio files (.wav), sound card

- Reading SoundFont files (instrumentdefinitions)


What have we done?



• Additional supporting infrastructure:

- Input: MIDI files (musical scores), keyboard

- Output: audio files (.wav), sound card

- Reading SoundFont files (instrumentdefinitions)

• Status: proof-of-concept, but decent performance.


Example 1: Sine oscillator

oscSine fcv

oscSine :: Frequency → SF CV Sample

oscSine f0 = proc cv → do

let f = f0 ∗ (2 ∗∗ cv)

phi ← integral−≺ 2 ∗ pi ∗ f

returnA−≺ sin phi

constant 0 ≫ oscSine 440


Example 2: Vibrato

0oscSine 5.0 oscSine f*0.05

constant 0

≫ oscSine 5.0

≫ arr (∗0.05)

≫ oscSine 440


Example 3: 50’s Sci Fi

0oscSine 3.0

oscSine f

*0.2

-0.25+1.0

+

sciFi :: SF () Sample

sciFi = proc ()→ do

und ← arr (∗0.2) ≪ oscSine 3.0−≺ 0

swp ← arr (+1.0) ≪ integral −≺ −0.25

audio ← oscSine 440 −≺ und + swp

returnA−≺ audioIntroduction to Functional Reactive Programming through Games and Music – p.38/50

Envelope Generators (1)

A D S R

key on key off t

envGen :: CV → [(Time,CV )]→ (Maybe Int)

→ SF (Event ()) (CV ,Event ())

envEx = envGen 0 [(0.5, 1), (0.5, 0.5), (1.0, 0.5), (0.7, 0)]

(Just 3)



How to implement?



How to implement?

Integration of a step function yields suitableshapes:

t

∫−→

A D S R

key on key off t



t

afterEach :: [(Time, b)]→ SF a (Event b)

hold :: a → SF (Event a) a

steps = afterEach [(0.7, 2), (0.5,−1), (0.5, 0), (1,−0.7), (0.7, 0)]

≫ hold 0



Envelope generator with predetermined shape:

envGenAux :: CV → [(Time,CV )]→ SF a CV

envGenAux l0 tls = afterEach trs ≫ hold r0

≫ integral ≫ arr (+l0 )

where

(r0 , trs) = toRates l0 tls


Example 4: Bell

0

*oscSine f*2.0oscSine (f*2.33)

envBell

bell :: Frequency → SF () (Sample,Event)

bell f = proc ()→ do

m ← oscSine (2.33 ∗ f )−≺ 0

audio ← oscSine f −≺ 2.0 ∗m

(ampl , end)← envBell −≺ noEvent

returnA−≺ (audio ∗ ampl , end)


Example 5: Tinkling Bell

tinkle :: SF () Sample

tinkle = (repeatedly 0.25 84

≫ constant ()

&&&arr (fmap (bell ◦midiNoteToFreq))

≫ rSwitch (constant 0))


Example 6: Playing a C-major scale

scale :: SF () Sample

scale = (afterEach [(0.0, 60), (2.0, 62), (2.0, 64),

(2.0, 65), (2.0, 67), (2.0, 69),

(2.0, 71), (2.0, 72)]

≫ constant ()

&&&arr (fmap (bell ◦midiNoteToFreq))

≫ rSwitch (constant 0))

&&&after 16 ()


Example 7: Playing simultaneous notes

mysterySong :: SF () (Sample,Event ())

mysterySong = proc → do

t ← tinkle −≺ ()

m ← mystery−≺ ()

returnA−≺ (0.4 ∗ t + 0.6 ∗m)


A polyphonic synthesizer (1)

Sample-playing monophnic synthesizer:• Read samples (instrument recordings) from

SoundFont file into internal table.• Oscillator similar to sine oscillator, except

table lookup and interpolation instead ofcomputing the sine.

SoundFont synthesizer structure:

Envelopes

LFO

Modulators

Oscillator Lowpass filter Amplifier

FrequencyReverb

ChorusVolume

Fc


A polyphonic synthesizer (2)

Exploit Yampa’s switching capabilities to:

• create and switch in a mono synth instance isresponse to each note on event;

• switch out the instance in response to acorresponding note off event.


Switched-on Yampa?


Switched-on Yampa?


Conclusions


Conclusions

• FRP offers one way to write interactive gamesand similar software in a declarative way.


Conclusions


• It allows systems to be described in terms ofwhole values varying over time.


Conclusions



• Not covered in this talk:


Conclusions




- Not everything fit easily into the FRPparadigm: e.g., interfacing to existing GUItoolkits, other imperative APIs.


Conclusions




- Not everything fit easily into the FRPparadigm: e.g., interfacing to existing GUItoolkits, other imperative APIs.

- But also such APIs can be given a “whole-valuetreatment” to improve the fit within a declarativesetting. E.g. Reactive Values and Relations.


Documents

Introduction to Functional Reactive Programming through ...webuser.uni-weimar.de/~xume0027/lecture02-frp.pdf · Introduction to Functional Reactive Programming through Games and Music