Analyzing Hadoop with Hadoop

  • Published on
    07-Jul-2015

  • View
    499

  • Download
    0

DESCRIPTION

Talk that I gave at Berlin Buzzwords 2012. It shows why Hive doesn't fit in the Hadoop No-SQL environment and some examples of what information we were able to extract from the Hadoop user mailing list and git logs.

Transcript

<ul><li> 1. Analyzing Hadoopwith HadoopMontag, 4. Juni 12</li></ul> <p> 2. Data Grows Faster Than Moores Law!Unstructured: 61.7% growth Structured: 21.8 % growth http://www.emc.com/about/news/press/2011/20110628-01.htm sg@datameer.com, condential - Do not distributeMontag, 4. Juni 12 3. 30+ Years WorkowSlow StaticBarrierBusiness ETLData Warehouse IntelligenceFast DynamicAgile Raw LoadHadoop Analytics sg@datameer.com, condential - Do not distributeMontag, 4. Juni 12 4. Hadoop + Hive SQL 10+M NO-SQL Hadoop LOC http://thepage.time.com/2009/04/18/why-is-this-elephant-crying/http://dearcomputer.nl/gir/?q=nerd+&amp;s=4&amp;b=Rip+Google!Montag, 4. Juni 12 5. Evolution backward 1970ANSI SQL ORM JDO NO-SQL Hive SEQUEL Structured English Query Language http://chelseavose.wordpress.com/2012/01/26/is-evolution-real/Montag, 4. Juni 12 6. Unstructured + StructuredMontag, 4. Juni 12 7. git log --numstat --pretty=format:%H,%ai,%cn,%ce%+BMontag, 4. Juni 12 8. Data Quality?Montag, 4. Juni 12 9. Results...Montag, 4. Juni 12 10. Commits per Year200Montag, 4. Juni 12 11. LOC Changes per Year 7,000,000Montag, 4. Juni 12 12. Most Lines Added 1,500,000Montag, 4. Juni 12 13. 2006 eMails vs Commitscommitsemails 72Montag, 4. Juni 12 14. 2011 eMails vs Commits commits emails559Montag, 4. Juni 12 15. EMails per Month800Montag, 4. Juni 12 16. Most Discussed, Least ChangedMontag, 4. Juni 12 17. Most Active Emailers 900Montag, 4. Juni 12 18. Were hiring!Montag, 4. Juni 12 19. Emails with Most RepliesMontag, 4. Juni 12 20. Avg Characters per Commit Message 120Montag, 4. Juni 12 21. Longest Comment 35,000Montag, 4. Juni 12 22. Email Activity per TimezoneMontag, 4. Juni 12 23. Follow us: @datameerMontag, 4. Juni 12</p>

Recommended

View more >