Embed Size (px)
Citation preview
Faster Coroutine Pipelines:A Reconstruction
Ruben Pieters
1
Tom Schrijvers
Stream Processing
2
Stream Processing
3Data
Stream Processing
4
Result
Data
Stream Processing
5
Result
IO
Data
Stream Processing
6
[Data] -> IO Result
Stream Processing
7
[Data] -> IO Result
Stream Processing
8
Stream Processing
9
Stream Processing
Data Result
IO
Stream Processing
10
Stream Processing
Data Result
IO
Stream Processing
11
Stream Processing
Data Result
IO
Stream Processing
12
Stream Processing
Data Result
IO
Stream Processing
13
Stream Processing
14
Stream Processing
15
Data Result
IO
Stream Processing
16
Traditional1,2
Alternative3,4
1Pipes, Gonzalez, Haskell Library2Conduit, Snoyman, Haskell Library3Continuations and Transducer Composition, Shivers and Might, PLDI ‘064Faster Coroutine Pipelines, Spivey, ICFP ‘17
Overview
17
Goal
18
Introduction to● Traditional Representation1,2
● Alternative Representation3,4
1Pipes, Gonzalez, Haskell Library2Conduit, Snoyman, Haskell Library3Continuations and Transducer Composition, Shivers and Might, PLDI ‘064Faster Coroutine Pipelines, Spivey, ICFP ‘17
Our Paper
19
Traditional1,2
Alternative3,4
1Pipes, Gonzalez, Haskell Library2Conduit, Snoyman, Haskell Library3Continuations and Transducer Composition, Shivers and Might, PLDI ‘064Faster Coroutine Pipelines, Spivey, ICFP ‘17
Systematic Derivation
Story
20
hellooooooo world!!!
I only just worked out that the Moon landings were a hoax
hellooooooo world!!!
im a nice person!
Step 1: Generate And Tweet
21
hellooooooo world!!!
I only just worked out that the Moon landings were a hoax
hellooooooo world!!!
im a nice person!
Step 2: Filter Bad
22
hellooooooo world!!!
I only just worked out that the Moon landings were a hoax
hellooooooo world!!!
im a nice person!
Step 3: Filter Duplicates
23
hellooooooo world!!!
I only just worked out that the Moon landings were a hoax
hellooooooo world!!!
im a nice person!
Problem While Filtering Duplicates
24
Step 1: Generate And Tweet
25
Producer/Consumer
26
Pr(oducer) o m r
Producer/Consumer
27
Pr(oducer) o m r
o o
Producer/Consumer
28
Pr(oducer) o m r
o o
Producer/Consumer
29
Pr(oducer) o m r
o or
Producer/Consumer
30
: Pr String IO Ø
o o=
Producer/Consumer
31
Co(nsumer) i m r
Producer/Consumer
32
Co(nsumer) i m r
i i
Producer/Consumer
33
Co(nsumer) i m r
i i
Producer/Consumer
34
Co(nsumer) i m r
i i r
Producer/Consumer
35
: Co String IO Ø
= i i
Merge
36
Merge
37
merge
Merge
38
a
a
a
a
Merge
39
a
a
a
a
Merge
40
a
a
a
a
Merge
41
a
a
a
a
Merge
42
a
a
a
a
Merge
43
a
a
a
a
Merge
44
a
a
a
a
Merge
45
a
a
a
a
Merge
46
a
a
a
a
Result Step 1
47
hellooooooo world!!!I only just worked out that the Moon landings were a hoax
Step 2: Filter Bad
48
Filter
49
filter
Pipe
50
P(ipe) i o m r
i
Pipe
51
P(ipe) i o m r
io
Pipe
52
P(ipe) i o m r
io
Pipe
53
= P(ipe) Ø o m rPr(oducer) o m r
Øo
Pipe
54
= P(ipe) i Ø m rCo(nsumer) o m r
iØ
Merge
55
a
ia
i
oa
a
o
Merge
56
a
ia
i
oa
a
o
Merge
57
a
ia
i
oa
a
o
Merge
58
a
ia
i
oa
a
o
i
Merge
59
a
i
oa
a
o
i
ai
Merge
60
a
i
oa
a
o
i
ai
Merge
61
a
i
oa
a
o
i
ai
o
Merge
62
a
i
oa
a
o
i
ai
oi
Merge
63
a
i
oa
o
i
ai
oi
a
Merge
64
a
i
oa
o
i
ai
oi
a
o
Merge
65
a
i
oa
o
i
ai
oi
a
o
Merge
66
a
i
oa
o
i
ai
oi
a
o
pipes: (>->)conduit: (.|)
Filter
67
filter
Filter
68
isBad i
isGood i
i
o
Filter
69
i
o
i
o
Filter
70
=filter filteri
o
Filter
71
a a
a a
a
a
a
a
filter
Filter
72
a
a a
a
a
a
a
filter
Result Step 2
73
Result Step 2
74
hellooooooo world!!!
Result Step 2
75
hellooooooo world!!!I only just worked out that the Moon landings were a hoax
Step 3: Filter Duplicates
76
Nub
77
nubfilter
Nub
78
ao
Nub
79
b
o
ao
/= a
Nub
80
b
o
c
o
ao
/= a
/= a&&/= b
Nub
81
= mergenub
filter(/= a)
io
nub
Result Step 3
82
nubfilter
Result Step 3
83
hellooooooo world!!!
I only just worked out that the Moon landings were a hoax
hellooooooo world!!!
im a nice person!
nubfilter
Result Step 3
84
Contains many merge calls
nubfilter
Merge Performance
85
Alternative Representation
86
Traditional Representation
87
o o
Pr o r = (o, … (o, Pr o r))
…
Traditional Representation
88
i i
Co i r = (λi. … (λi. Co i r))
…
Alternative Representation
89
o o
Pr o r: Ko -> r =
…
Alternative Representation
90
o o
Pr o r: Ko -> r =
λk. k o ( … (λk. k o ( Pr o r)))
Callback
…
Alternative Representation
91
o o
Pr o r: Ko -> r =
λk. k o ( … (λk. k o ( Pr o r)))
…
Alternative Representation
92
i i
Co i r: Ki -> r =
λk. k (λi k. … (λi. Co i r))
…
Alternative Representation
93
merge = f xwhere f = x =
i ii
o oo
Alternative Representation Pipe
94
io
?
Alternative Representation Pipe
95
io
P i o r: Ko -> K
i -> r =
Alternative Representation Pipe
96
io
P i o r: Ko -> K
i -> r =
λko k
i. k
i (λi k
o k
i. k
o o
(P i o r))
Alternative Pipe Merge
97
mergeP = f xwhere f :
x : Ki -> K
a -> r
Ka -> K
o -> r
mergeP = f xwhere f :
x : Ki -> K
a -> r
Ka -> K
o -> r
Alternative Pipe Merge
98
a
Alternative Pipe Merge
99
mergeP = ? p qwhere p = q =
i
o
a
Alternative Pipe Merge
100
mergeP = _ : Ki -> K
o -> r
where p : K
i -> K
a -> r
q : K
a -> K
o -> r
Alternative Pipe Merge
101
mergeP = _ : Ki -> K
o -> r
where p : K
i -> K
a -> r
q : K
a -> K
o -> r
Alternative Pipe Merge
102
mergeP = λko k
i. _ : r
where p : K
i -> K
a -> r
q : K
a -> K
o -> r
Alternative Pipe Merge
103
mergeP = λko k
i. _ : r
where λx. p k
i x :
K
a -> r
λx. p x k
o : K
a ->
r
Alternative Pipe Merge
104
mergeP = merge p’ q’where p’ : q’ :
i ii
o oo
Alternative Pipe Merge
105
mergeP = p’ q’where p’ : q’ :
i ii
o oo
Alternative Pipe Merge
106
Nested Traditional Merge
107
i
a
a
aa
a
a
Nested Traditional Merge
108
i
a
a
aa
a
a
Nested Traditional Merge
109
i
a
a
aa
a
a
Nested Traditional Merge
110
i
a
a
aa
a
a
a
Intermediate de- and reconstruction
aa
a
Nested Traditional Merge
111
i
a
a
aa
a
a
aa
Intermediate de- and reconstruction
Nested Traditional Merge
112
i
a
a
aa
a
a
aa
Intermediate de- and reconstruction
ia
Nested Traditional Merge
113
i
a
a
aa
a
a
a
Intermediate de- and reconstruction
i
Nested Traditional Merge
114
i
a
a
aa
a
a
i
a
a
Nested Alternative Merge
115
i
a
a
aa
a
a
Pass the callback through the nested merge
Nested Alternative Merge
116
i
a
a
aa
a
a
Pass the callback through the nested merge
Nested Alternative Merge
117
i
a
a
aa
a
a
Pass the callback through the nested merge
Our Paper
118
How can we derive the fast representation from the traditional one systematically?
[email protected] images from Machinarium
