View
418
Download
1
Category
Preview:
Citation preview
FS2for
Fun & Profit
@adilakhter
1
The next 45 minutes
... is about FS2
2
@ load.ivy("co.fs2" %% "fs2-core" % "0.9.0-M3")
@ load.ivy("co.fs2" %% "fs2-io" % "0.9.0-M3")
3
4
5
fs2.util.
Task
6
Taskis a
Monadic context for Asynchronous computation
7
Task[A]
8
Taskseparates
what to do from when to do
9
@ def launchMissile() = ... // side-effectlaunchMissile: ()Unit
10
@ val launchFuture = Future { launchMissile() }launchFuture: scala.concurrent.Future[Unit] = ...// A missile has already launched!
11
@ launchFuture// nothing happens due to side-effect memoization!
12
@ val launchTask = Task { launchMissile() }launchTask: fs2.util.Task[Unit]= ...
13
@ launchTask.unsafeRun // <- a missile now launched
14
@ launchTask.unsafeRun // missile 1 -> launched@ launchTask.unsafeRun // missile 2 -> launched
15
Taskseparates
what to do from when to do
16
Taskis
purely functional & compositional
17
Task Combinators
4 now[A](a: A): Task[A]
4 delay[A](a: => A): Task[A]
4 fail(e: Throwable): Task[Nothing]
4 fork[A](a: => Task[A]): Task[A]
18
Example: Retrieve Tweets
@ import twitter4j._
@ def statuses(query: Query): Task[List[Status]] = Task { twitterClient.search(query).getTweets.toList}
19
FS2
20
Goals
4 Incremental IO & Stream Processing
4 Modularity & Composability
4 Resource-safety
4 Performance
21
Goals
4 Incremental IO & Stream Processing
4 Modularity & Composability
4 Resource-safety
4 Performance
22
@ import fs2._@ import fs2.util._
@ implicit val strategy: Strategy = Strategy.fromFixedDaemonPool(8)@ implicit val scheduler: Scheduler = Scheduler.fromFixedDaemonPool(8)
23
File IO using FS2import java.nio.file.Paths
def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0)
val streamConverter: Task[Unit] = io.file.readAll[Task](Paths.get("testdata/fahrenheit.txt"), 4096) .through(text.utf8Decode) .through(text.lines) .filter(s => !s.trim.isEmpty && !s.startsWith("//")) .map(line => fahrenheitToCelsius(line.toDouble).toString) .intersperse("\n") .through(text.utf8Encode) .through(io.file.writeAll(Paths.get("testdata/celsius.txt"))) .run
streamConverter.unsafeRun()
24
FS2
... provides an abstraction to declaratively specify how to obtain stream of data.
25
Stream[F[_], O]
26
27
28
29
Example: Pure Streams
@ Stream(1,2,3)res29: Stream[Nothing, Int] = Segment(Emit(Chunk(1, 2, 3)))
@ Stream(1,2,3) ++ Stream(4,5,6)res30: Stream[Nothing, Int] = append(Segment(Emit(Chunk(1, 2, 3))), Segment(Emit(Chunk(()))).flatMap(<function1>))
30
Stream ≡ Emit | Await | Done
31
Emit
Emit[F[_],O]( head: Seq[O], tail: Stream[F, O] = Done)
32
Example: Pure Streams
@ import Stream._import Stream._
@ emit(1) // emits valueres30: Stream[Nothing, Int] = Segment(Emit(Chunk(1)))
@ emits(Seq(1,4,10,20)) // <- emits sequenceres31: Stream[Nothing, Int] = Segment(Emit(Chunk(1, 4, 10, 20)))
33
Await
Await[F[_], I, O]( req: F[I], // side-effect recv: I ⇒ Stream[F, O]) // continuation of Stream.
34
Example: Effectful Stream
@ def statuses(query: Query): Task[List[Status]] = ...
35
// Builds a Twitter Search Process for a given Query@ def tweetStream(query: Query): Stream[Task, Status] = { val queryTask: Task[List[Status]] = statuses(query) eval(queryTask).flatMap { statusList ⇒ emitAll(statusList) }}
36
//In essence, it builds: Stream: Await ⇒ (_ ⇒ Emit ⇒ Done)
val searchStream = tweetStream(new Query("#spark"))
37
Example: A Stream of Positive Integers
@ import Stream._import Stream._
@ def integerStream: Stream[Task, Int] = { def next (i: Int) : Stream[Task, Int] = eval(Task(i)).flatMap { i ⇒ emit(i) ++ next(i + 1) } next(1) }// in essence, it describes Stream of {1, 2, 3, 4 ....}
38
Stream[F, O]
... represents a stream of O valueswhich can interleave external requeststo evaluate computation of form F[_]
39
Stream Combinators
4 Stream.constant
4 Stream.eval
4 Stream.repeatEval
4 Stream.evalMap
4 fs2.io.readAll
4 ...40
Evaluating a Stream
41
Stream.run
@ integerStream.take(10)res23: Stream[Task, Int] = ...
@ integerStream.take(10).runres24: Task[Unit] = Task
@ integerStream.take(10).run.unsafeRun
42
Stream.runLog
scala> integerStream.take(10).runLog.unsafeRunres4: Vector[Int] = Vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
43
Stream.runLog
What does happen when we execute integerStream.runLog.run?@ def integerStream: Stream[Task, Int] = { def next (i: Int) : Stream[Task, Int] = eval(Task(i)).flatMap { i ⇒ emit(i) ++ next(i + 1) } next(1) }
@ integerStream.runLog.unsafeRun // <--- ?
44
Other Run Combinators
> def runFree: fs2.util.Free[F,Unit]
> def runFold[B](z: B)(f: (B, A) => B): F[B]
45
Transformations
46
Stream.map
@ val stream = integerStream.map(_ * 100)
stream: Stream[Task, Int] =attemptEval(Task).flatMap(<function1>).flatMap(<function1>).mapChunks(<function1>)
@ stream.take(5).runLog.unsafeRunres39: Vector[Int] = Vector(100, 200, 300, 400, 500)
47
Stream.flatMap
@ integerStream.flatMap { i => emits(List.fill(i)(i)) } .take(10) .runLog .unsafeRun
res45: Vector[Int] = Vector(1, 2, 2, 3, 3, 3, 4, 4, 4, 4)
48
Stream.zip
@ val zippedStream = integerStream zip emitAll(Seq("A", "B"))res52: Stream[Task, (Int, String)] =evalScope(Scope(Bind(Eval(Snapshot),<function1>))).flatMap(<function1>)
@ zippedStream.runLog.unsafeRunres54: Vector[(Int, String)] = Vector((1, "A"), (2, "B"))
49
Composability
50
Pipe
> Stream[F, A].through(Pipe[A, B]) => Stream[F, B]
51
Pipe
type Pipe[F[_],-I,+O] = Stream[F,I] => Stream[F,O]
52
Pipe
@ import pipe._import pipe._
53
Example
@ integerStream |> filter( _%2 == 0)res59: Stream[Task, Int] = ...
54
Example
@ integerStream |> filter( _%2 == 0) |> exists(_ == 10)res60: Stream[Task, Boolean] = ...
55
Pipe2
type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2]) => Stream[F,O]
56
Sink
> Stream[F, A].to(Sink[F, A]) => Stream[F, Unit]
57
Sink
type Sink[F[_],-I] = Pipe[F,I,Unit]
58
Example
@ def log[A]: Sink[Task, A] = _.evalMap{ x => Task{ println(s"$x") } }
59
Example
scala> val streamWithSink: Stream[Task, Unit] | = integerStream.take(5) to printIntsscala> streamWithSink.run.unsafeRun12345
60
ExampleSentiment Analysis of Tweets
61
62
Building Source
@ def src: Stream[Task, Query] = awakeEvery[Task](15 seconds).map { _ ⇒ new Query("#spark") }
63
Twitter Query Pipe
// defined earlierdef statuses(query: Query): Task[List[Status]] = ...
def twitterPipe: Pipe[Task, Query, List[Status]] = _.evalMap{ query ⇒ statuses(query) }
64
Stream Pipeline
src .through(twitterPipe) // Stream[Task, List[Status]] .flatMap(Stream.emits(_)) // Stream[Task, Status] .map(s ⇒ Tweet(s)) // Stream[Task, Tweet]
65
Sentiment Analysis Pipe
@ import SentimentAnalyzer._
@ def analyze(t: Tweet): Task[EnrichedTweet] = Task { EnrichedTweet( t.author, t.body, t.retweetCount, sentiment(t.body)) }
66
@ def analysisPipe: Pipe[Task, Tweet, EnrichedTweet] = _.evalMap{tweet ⇒ analyze(tweet)}
67
Stream Pipeline (so far)
src .through(twitterPipe) // Stream[Task, List[Status]] .flatMap(Stream.emits(_)) // Stream[Task, Status] .map(s ⇒ Tweet(s)) // Stream[Task, Tweet]
68
Stream Pipeline
src .through(twitterPipe) // Stream[Task, List[Status]] .flatMap(Stream.emits(_)) // Stream[Task, Status] .map(s ⇒ Tweet(s)) // Stream[Task, Tweet]
.through(analysisPipe) // Stream[Task, EnrichedTweet]
69
Stream Pipeline
@ val tweetStream = src .through(twitterPipe) // Stream[Task, List[Status]] .flatMap(Stream.emits(_)) // Stream[Task, Status] .map(s ⇒ Tweet(s)) // Stream[Task, Tweet]
.through(analysisPipe) // Stream[Task, EnrichedTweet]
70
Evaluating Stream Pipeline
@ val consoleTweetStream = tweetStream.to(snk) // Stream[Task, Unit]
@ consoleTweetStream.run.unsafeRun
71
72
Connecting Stream with WS
// http4s Websocket Routecase r @ GET -> Root / "websocket" ⇒ val query = new Query("#spark") val stream = tweetStream(query).map(Text(_))
WS(Exchange(stream, ...)) // <- connected stream to WS
73
74
Merge & Join
75
@ val s = integerStream .through(randomDelays(1.second)) .through(log("s"))
@ val t = range(1, 3) .through(randomDelays(1.second)) .through(log("t"))
76
Interleave
@ s interleave t res89: Stream[Task, Int] = ...
@ (s interleave t).through(log("interleaved")).runLog.unsafeRun
s > 1t > 1interleaved > 1interleaved > 1s > 2t > 2interleaved > 2interleaved > 2s > 3res90: Vector[Int] = Vector(1, 1, 2, 2)
77
Merge@ (s merge t).through(log("merged")).runLog.unsafeRuns > 1merged > 1t > 1merged > 1s > 2merged > 2s > 3merged > 3t > 2merged > 2s > 4merged > 4s > 5merged > 5....
78
Either
@ s either tres22: Stream[Task, Either[Int, Int]] = ...
@ Stream.emit("A, B, C") either tres23: Stream[Task, Either[String, Int]]
79
Join
@ val u = Stream.range(10, 20)u: Stream[Nothing, Int] = ...@@ val streams: Stream[Task, Stream[Task, Int]] = Stream(s, t, u)streams: Stream[Task, Stream[Task, Int]] = ...
@ concurrent.join(3)(streams)res27: Stream[Task, Int] = ...
80
“There are two types oflibraries: the ones people hateand the ones nobody use"
— Unknown
81
Thank You
82
CreditsCredits for the images used in slides: Mario Sixtus.
References and Further Readings4 FS2: The Official Guide
4 Scalaz-Stream Masterclass by Runar at NE Scala 2016
4 Scalaz Task - the missing documentation
4 scalaz-stream User Notes
4 Comparing akka-stream and scalaz-stream with code examples
4 Additional Resources
83
Recommended