Scala - where are are - where we plan to go

Scala- where are are

- where we plan to go

Martin Odersky

Welcome to ScalaDays!

• 155 attendees, coming from all corners of the planet.• We are packed. Sadly, many more people could not be

admitted for lack of space.• This shows the drive and energy of this community!• The Scala team is looking forward to talk with you and

learn from you during the next two days.• We are particularly glad to have you here at EPFL, the

place where Scala was born.

History (incomplete)

2001 Scala design begins, as a more practicalsuccessor to Funnel.

2003 First experimental release, first coursetaught with it.

2004 Scalable Component Abstractions paper.2005 Scala 2.0, written in Scala.2006 Industrial adoption starts.2007 First Lift release.2008 First Scala LiftOff, Twitter adopts Scala.2009 Big increase in adoption, IDEs mature.2010 First ScalaDays.

The main idea behind Scala

Scala is a Unifier

Agile, with lightweight syntax

Object-Oriented Scala Functional

Safe and performant, with strong static typing

The Philosophy behind Scala

Put productivity and creativity back in the hands of developers.

“Joy is underrated as a metric for a language’s potential success in a development organization.” a3lx@twitter

- address professional developers

- trust them & value their expertise

- (don’t tell them how they should do their job)

Some users in industry

Open Source Ecosystem

Many great projects, including:

lift (David P. & friends)akka (Jonas B.)kestrel, cachet, queroulous (Twitter)migrations (Sony)sbt (Mark H.)ScalaTest (Bill V. & friends)specs, ScalaCheck, ScalaModules, ScalaQL, ....

( the list goes on, many more tools & projects presented at this conference )

Now: 2.8

First release candidate Scala 2.8 RC1 is out!

This is primarily a consolidating release• Some new language features• More complete libraries• Redesigns of a few features that did not

work out well.• Big improvements in tooling.• Big improvements in stability.• Laying the groundwork for the future.

Scala 2.8

named and default parameters

copy operation

nested annotations

package objects

safer package nesting

@specialized

new collection API

vectors

hash tries

new treatment of arrays and strings

manifests

better implicits

better type inference

continuations

packrat parsing

faster actors

improved Swing support

better IDEs

interactive compiler API

new scaladoc

better REPL

Named and Default Parameters

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

def newTable(size: Int = 100, load: Float = 0.5f) =

new HashMap[String, String]{

override val initialSize = size

override val loadFactor = (load * 1000).toInt

}

}

newTable(50, 0.75f)

newTable()

newTable(load = 0.5f)

newTable(size = 20, load = 0.5f)

Defaults can be arbitrary expressions, not just constants.Default evaluation is dynamic.

Copy OperationScala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Problem: Case classes are great for pattern matching,but how do you transform data?

Often need to update (functionally) some specific field(s)of a date structure defined by case classes.

Solution: Selective copy method.

Q: How to define “copy”?

case class Tree[+T](elem: T, left: Tree[T], right: Tree[T])

def incElem(t: Tree[Int]) = t copy (elem = t.elem + 1)

Defining “copy”Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Problem: A class with n parameters allows 2n possible combinations of parameters to copy.

Default parameters to the rescue.

case class Tree[+T](elem: T, left: Tree[T], right: Tree[T]) {

// generated automatically:

def copy[U](elem: U = this.elem,

left: Tree[U] = this.left,

right: Tree[U] = this.right) =

Branch(elem, left, right)

}

Nested Annotations

• Scala 2.7 largely followed Java annotation syntax, but left out some of the more convoluted constructions. Nested annotations were not possible.

• Scala 2.8 throws out (deprecates) all special annotation syntax.

• Instead uses class constructors, which can have named parameters.

• This greatly simplifies the syntax, and allows all annotations to be expressed, including nested ones.

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

@JoinTable( joinColumns = Array(new JoinColumn(name = "a name“)) var test: String = _

@specializedProblem: Generics cause a performance hit because they

require a uniform representation: everything needs to be boxed.

Example:

Scala 2.8


copy operation

nested annotations

@specialized

new collection API

vectors

hash tries


package objects


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

trait Function1[-T, +U] { def apply(x: T): U }

def app(x: Int, fn: Int => Int) = fn(x)

app(7, x => x * x)Translatedby scalac 2.7 to: trait Function1 {

def apply(x: Object): Object }

class anonfun extends Function1 { def apply(x: Object): Object = Int.box(apply(Int.unbox(x))) def apply(x: Int): Int = x * x}

def app(x: Int, fn: Function1) = Int.unbox(fn(Int.box(x)))

app(7, new anonfun)

If Function1 is @specialized ...Scala 2.8


copy operation

nested annotations

@specialized

new collection API

vectors

hash tries


package objects


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

trait Function1[@specialized -T, @specialized +U] { def apply(x: T): U }

def app(x: Int, fn: Int => Int) = fn(x)app(7, x => x * x)

... scalac 2.8 generates instead:

trait Function1 { def apply(x: Object): Object def apply$$II(x: Int): Int = Int.unbox(apply(Int.box(x))) // specializations for other primitive types}class anonfun extends (Int => Int) { def apply(x: Object): Object = Int.box(applyIntInt(Int.unbox(x))) override def apply$$II(x: Int): Int = x * x}

def app(x: Int, fn: Function1) = fn.apply$II(x)

app(7, anonfun)

Scala CollectionsScala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Collection Properties:object-oriented

generic: List[T], Map[K, V]

optionally persistent, e.g. collection.immutable.Seq

higher-order, with methods such as foreach, map, filter.

Uniform return type principle: Operations return collections of the same type (constructor) as their left operand.

scala> val ys = List(1, 2, 3)ys: List[Int] = List(1, 2, 3)

scala> val xs: Seq[Int] = ysxs: Seq[Int] = List(1, 2, 3)

scala> xs map (_ + 1)res0: Seq[Int] = List(2, 3, 4)

scala> ys map (_ + 1)res1: List[Int] = List(2, 3, 4)

Old Collection StructureScala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

trait Iterable[A] { def filter(p: A => Boolean): Iterable[A]

= ... def partition(p: A => Boolean) = (filter(p(_)), filter(!p(_))) def map[B](f: A => B): Iterable[B] = ...}

trait Seq[A] extends Iterable[A] { def filter(p: A => Boolean): Seq[A] = ... override def partition(p: A => Boolean) = (filter(p(_)), filter(!p(_))) def map[B](f: A => B): Seq[B] = ...}

Types force duplication!

New Collection API

• Avoids type & code duplication through– uniform framework of traversers and builders.

– supported by: implicits, higher-kinded types

• Uniform package organization reflects mutability:– scala.collection– scala.collection.mutable– scala.collection.immutable

• Richer and more regular set of collection operations.

• New collection implementations: vectors and hash tries.

• More info: “Fighting bit rot with types”, FSTTCS09

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Vectors and Hash Tries

• Trees with branch factor of 32.• Persistent data structures with very efficient sequential

and random access.• Invented by Phil Bagwell, then adopted in Clojure.• New: Persistent prepend/append/update in constant

amortized time.• Next: Fast splits and joins for parallel transformations.

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Package Objects

Motivation:• The new collection design reshuffled a lot of

classes.• For instance:

scala.List scala.collection.immutable.List

• How can we keep the old aliases around?• How can we migrate old to new?• Easy: put

type List = scala.collection.immutable.Listval List = scala.collection.immutable.List

into the scala package.• But how to we put a type alias and a val definition

into a package?

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Package ObjectsScala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Problem: Everything has to live inside an object or class.=> No top-level value definitions, type aliases, implicits.

Package objects make packages more like objects.

They also let you define your own ``Predefs’’.

package object scala {

type List[+T] = scala.collection.immutable.List

val List = scala.collection.immutable.List

...

}

Safer Package Nesting

• Java knows only absolute packages• But Scala’s packages nest• This can lead to an impedance mismatch.• For instance:

package net.liftwebjava.lang.System.print(“hello”)

• This used to fail if net contains a subpackage java.• New rules: Intermediate packages only visible in

multiple package clauses.• To access net.java as java, you’d need to write

package netpackage liftweb...

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

New Treatment of ArraysScala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

• Arrays were “magical” up to 2.8– sometimes primitive– sometimes wrapped– automatic boxing/unboxing conversions based

on static type.

• This could be confusing and slow.• 2.8 removes the magic.• Arrays are always Java arrays• Two implicit conversions make sure that

– arrays are compatible with sequences.– arrays support all sequence operations

• Strings are treated similarly to arrays.

Manifests

• Question: How to create an Array[T] where T is a type parameter?

• No magic allowed!• Solution: supply the run-time type as a type

descriptor called a Manifest or ClassManifest:

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

def evens[T: ClassManifest](xs: Vector[T]): Array[T] = { val arr = new Array[T]((xs.length + 1) / 2) for (i <- 0 until xs.length by 2) arr(i / 2) = xs(i) arr}

def evens[T](xs: Vector[T]) (implicit ev: ClassManifest[T]): Array[T]

The first line is equivalent to:

Better ImplicitsBetter Type Inference

• Type inference for implicits has been refined.

• Previously: Infer type first, search implicit for inferred type afterwards.

• Now: Implicit search can drive type inference.

• This makes implicits more powerful.• Other advance: Type constructors of

higher-kinded types can now be inferred.

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Delimited Continuations

• Continuations capture the return stack as a data structure.

• Delimited continuations capture only up to some enclosing scope.

• Lost of advanced control structures can be built on them, e.g:– coroutines– event-based I/O– reactive programming– lightweight actors

• Scala 2.8 supports delimited continuations in a compiler plugin.

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


delimited continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Better IDEs

• The Scala plugins for Eclipse, IntelliJ, Netbeans all have matured a lot.

• The Eclipse and Netbeans plugins are now based on completely redesigned compiler interfaces for command completion, semantic highlighting and incremental building.

• If you have discounted IDEs before for Scala, maybe now is the time to try again!

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Docs for Advanced Types

Problem: Advanced statically typed APIs pose challenges for presentation and documentation.

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Simpler Type Views

• How to explaindef map[B, To](f: A => B)

(implicit cbf: CanBuildFrom[Coll, B, To]): To

to a beginner?• Key observation: We can approximate the type of map.• For everyone but the most expert user

def map[B](f: A => B): Traversable[B] // in class Traversabledef map[B](f: A => B): Seq[B] // in class Seq, etc

is detailed enough.• These types are correct, they are just not as general as

the type that’s actually implemented.• These are documented in the new ScalaDoc as use

cases.

Scala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

New ScalaDocScala 2.8


copy operation

nested annotations

package objects


@specialized

new collection API

vectors

hash tries


manifests

better implicits


continuations

packrat parsing

faster actors


better IDEs


new scaladoc

better REPL

Why 2.8, not 3.0?

• Scala 2.8 is numerically a point release, but feels like a major release.

• So why did we not name it Scala 3.0?

• 2.8 has been 18 months in the making, and acquired more changes on the

way.

• Originally, we just wanted to rewrite collections.

• That took longer than expected.

• Meanwhile, a lot more additions and changes got included.

• But 2.8 was already announced, and referred to in books and other

documentation.

• In light of this, we judged it too confusing to change the version number in

mid-flight.

And now...

...what’s next?

Next: Scala on .NET

• Work started again to have a native Scala compiler on .NET with Visual Studio integration.

• Funded my Microsoft.• Stay tuned...

Next: Reliability and Contracts

• Lost of things happening around contracts and verification.

• Peter Müller: Scala verifier• Viktor Kuncak: Scala code synthesis from constraints• Our group: effects• Common project: ProgLab.NET

Next: Concurrency

• Actor encapsulation– How to ensure that actors do not share memory?– Solved by capability-based uniqueness type system.– Implemented in compiler plugin.– Paper in ECOOP 2010.

• Software Transactional Memory– CCSTM work in this conference

• Open question: What’s a good combination of actors and transactions?

Next: Parallelism

• PPP Challenge: How to make use of multi-cores, GPUs, clusters, clouds, to achieve better performance for normal applications?

• First step in this direction: Parallel collections.– will be integrated in the new collection API.

– based on persistent data structures with fast splits and joins.

• Longer-term vision: Embedded parallel DSLs.

• See keynote by Kunle Olokutun tomorrow.

Next: The Green House

Great work on Scala is springing up everywhere, no longer focussed just at EPFL.

Q: How can we integrate contributions?Tension: Want to get lots of exciting

new stuff contributed, yet maintain a stable and reliable distribution.

Planned Solution: three-tiered:Incubator Hosts collaborative Scala

projectsGreen house A “cutting edge” version of

the distribution with experimental features.

Trunk The stable distribution.

Thanks

• To the Scala team at EPFL:Lukas Rytz, Hubert Plociniczak, Iulian Dragos, Miguel Garcia, Gilles Dubochet, Philipp Haller, Aleksandar Prokopec, Antonio Cunei, Tiark Rompf, Donna Malayeri, Phil Bagwell, Adriaan Moors, Ingo Maier.

• To our external committers and frequent contributors:Paul Phillips, Miles Sabin, Ilya Sergey, Caoyuan Deng, James Matlik, Frank Teubler, Kevin Wright, Manohar Jonnalagedda, Pedro Furlanetto, Johannes Rudolph, Jason Zaugg, Seth Tisue, Ismael Juma, Mark Harrah, Colin Howe, Mirko Stocker, Spiros Tzavellas, Matt Russell, David Taylor

• To the community at large for your support, suggestions, encouragements.

Documents

Scala - where are are - where we plan to go