Making better news with Spark

Phil Wills@philwills

What problems were we trying to solve?

What issues have we encountered?

What’s worked well?

What problems were we trying to solve?

9

Why Spark?

What issues have we encountered?

ExecutorLostFailure (executor lost)

object Foo { def bar(i: Int) = i + 1 def go() = sc.parallelize(List(1,2,3)) .map(bar(_)) .toArray()

}

Foo.go()

Task not serializable

object Foo2 { val bar = (i: Int) => i + 1 def go() = sc.parallelize(List(1,2,3)) .map(bar) .toArray()

}

Foo2.go()

Array[Int] = Array(2, 3, 4)

© Copyright Albert Bridge

--num-executors--executor-cores--executor-memory--driver-cores--driver-memory

What’s worked well?

Simple to use ML

Ad-hoc querying

Bulk transformations

Questions?

@philwills