Upload
claus
View
56
Download
2
Embed Size (px)
DESCRIPTION
Making your Apps Smarter with Azure HDInsight. Matt Winkler, @ mwinkle Principal Program Manager 3-529. Agenda. What is HDInsight ? Programming Hadoop Integrating with your apps. Windows Azure HDInsight. On demand Apache Hadoop Clusters. Elastic – what you want, when you want it - PowerPoint PPT Presentation
Citation preview
Making your Apps Smarter with Azure HDInsightMatt Winkler, @mwinklePrincipal Program Manager3-529
What is HDInsight?Programming HadoopIntegrating with your apps
Agenda
Elastic – what you want, when you want itSimple – 3 clicks, 10 minutes to HadoopSecure – isolated, secured by default
Big data processing on top of Azure Storage
On demand Apache Hadoop Clusters
Windows Azure HDInsight
Demo: HDInsight 0 – 60
Storage + proliferation of compute models for data processing at scaleBegan life as an open source implementation of Google’s Map/Reduce and GFS papersIn use at many major web companies at massive scale (1000’s of node, PB’s of storage)
“Hadoop is a distributed system for counting words.”
scalding readme
Hadoop there it is
Drill
Pig
Map/Reduce
Oozie
Sqoop
MahoutPegasus
Storm
Scalding
Cascading
Hive
Scoobi
Cloud as global aggregation point for sensors & devicesData born in the cloud in apps and servicesIn your app, what data are you not collecting? Why? What could you do with it?
“we should stop acting as if our goal is to author extremely elegant theories, and instead embrace complexity and make use of the best ally we have: the unreasonable effectiveness of data. ”
Peter Norvig, et al
But I don’t have big data!
Acquire – Compute – Publish – Consume
Acquire Publish
Azure BlobStorage
Shape
QueryExperiment
Acquire Publish
Azure BlobStorage
Shape
QueryExperiment
Consume
Authoring Jobs App Integration
Core Hadoop
Consistent REST API’s
Breadth of Clients (Java, JS, .NET, etc)
Authoring frameworks and languages
End User Tooling (IDE’s, Analyst tools, Command lines)
ConnectivityProgrammabilitySecurityLoosely coupled
LightweightLow cost to extendScenario oriented
Innovation flows upward
New compute modelsPerf enhancements
Extend breadth & depthEnable new scenariosIntegrate with current tool chains
App/Services writing to BlobData available via servicesPush to blob (on-prem => cloud)Copy To Blob
Acquire
No one tool to rule them all
Preprocessing – Cleansing / Shaping / EnrichingTraditional Analytics – Query Advanced Analytics – Experiments / Models
SDK & Command line tools for submitting and managing these jobs
Compute
Back to BlobsInto reporting engineInto “online” store – SQL, Mongo, etcAs another table
Publish
Data ExplorerODBC/JDBCREST API’sDirect from Blob StorageAdditional Hadoop jobs
Consume Results
Scheduled executionMore sophisticated analysisAnalyze impact of changesIncorporate other data sources
What Next?
HDInsight StatusAzure Public PreviewAvailable in 2 Azure RegionsDiscounted compute pricingSDK & CLI tools available
HDInsight, What’s Next? Committed SLAGlobally availableContinuing to improve Hadoop (including Stinger)Continued Investments in Cluster Configuration, Management, Integration and Developer tooling
Resourcesbig data on azurehdinsight getting startedhadoop sdk for .net (includes PowerShell for now) node.js sdk x-plat cli
Evaluate this session
Scan this QR code to evaluate this session and be automatically entered in a drawing to win a prize!
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.