Raising the Tides: Open Source Analytics for Data Science

Raising the Tides: Open Source Analytics for Data ScienceWes McKinney @wesmckinn

N E W S W E E K A I & D A T A S C I E N C E C O N F E R E N C E – C A P I T A L M A R K E T S2 M A R C H 2 0 1 7

Wes McKinney @wesmckinn

Me


Important Legal Information• The information presented here is offered for informational purposes only

and should not be used for any other purpose (including, without limitation, the making of investment decisions). Examples provided herein are for illustrative purposes only and are not necessarily based on actual data. Nothing herein constitutes: an offer to sell or the solicitation of any offer to buy any security or other interest; tax advice; or investment advice. This presentation shall remain the property of Two Sigma Investments, LP (“Two Sigma”) and Two Sigma reserves the right to require the return of this presentation at any time.

• Some of the images, logos or other material used herein may be protected by copyright and/or trademark. If so, such copyrights and/or trademarks are most likely owned by the entity that created the material and are used purely for identification and comment as fair use under international copyright and/or trademark laws. Use of such image, copyright or trademark does not imply any association with such organization (or endorsement of such organization) by Two Sigma, nor vice versa.

• Copyright © 2017 TWO SIGMA INVESTMENTS, LP. All rights reserved


In the next 20 minutes

∞ Important trends in the industry∞ Two Sigma involvement in open

source∞ Growing the community

WHAT I’M SEEING TODAY


Industry giants open source core AI and machine learning technology


Open source “disruption” in data science languages and supporting technologies


Observation #1:

User Mindshare is a Key Asset


Observation #2:

Tools may be less important than human capital and data


Two SigmaBuilding a state-of-the-art, collaborative data science platform


Scaling data science in many dimensions

∞ Access to diverse data sets



∞ Access to diverse data sets∞ Enhancing individual productivity



∞ Access to diverse data sets∞ Enhancing individual productivity∞ Computational capabilities: larger

and more complex data sets



∞ Access to diverse data sets∞ Enhancing individual productivity∞ Computational capabilities: larger

and more complex data sets∞ Collaboration within and across

teams

TOOLS AND THE“DATA SCIENTIST SHORTAGE”

WHY WE PARTICIPATE IN OPEN SOURCE


Why we participate in Open Source1. Drive progress and innovation in

foundational technologies



foundational technologies2. Increase the overall value,

interoperability, and sustainability of our closed source systems



foundational technologies2. Increase the overall value,

interoperability, and sustainability of our closed source systems

3. Raise awareness of problems faced at scale on real world data



foundational technologies2. Increase the overall value, interoperability,

and sustainability of our closed source systems


4. Benefit sooner from open source innovations


Why we participate in Open Source1. Drive progress and innovation in foundational

technologies2. Increase the overall value, interoperability,

and sustainability of our closed source systems


4. Benefit sooner from open source innovations5. Attract and retain the best engineering talent


Where we are investing

Collaboration and Publishing

Cluster Resource Management

Scalable / Distributed Computing

High Performance Data Processing


Core data infrastructure technologies

Apache Arrow

Apache Parquet

• Efficient columnar in-memory data processing

• High-speed, interoperable data messaging for Java, C++, Python

• Industry-standard columnar file format for distributed storage

• Efficient IO for Spark, Python, etc.


Open source in-memory and distributed analytics

• Popular Python analytics library

• Powerful and easy-to-use data cleaning, analytics, and time series processing

• Flint: scalable time series analytics for Spark

• Enhanced Python integration


Cluster resource management

• Scalable cluster resource manager

• Native container support

• Fair job scheduler for Mesos

• Managing multi-tenant Spark clusters

cook


Collaboration and publishing

• Notebook “kernels” for polyglot research and development

• Inter-language data exchange

• Leading web notebook & reproducible research development platform

• Interactive widgets framework

TOWARD HIGH TIDE:Preserving competitive advantage and building common knowledge

Thank youWes McKinney @wesmckinn

Technology

Raising the Tides: Open Source Analytics for Data Science