Click here to load reader

Big data, huh

  • View
    6

  • Download
    0

Embed Size (px)

Text of Big data, huh

  1. 1. r , . miaBIG DATA,HUH? CLARIFYING A SUBJECT"'71
  2. 2. ~ A.Lot of Tools ~ A Lot of Requirements* A Lot of Options
  3. 3. i Data creates meaning& ut how? 77-1- .. _ ,--~ V var-s --g-Tg;, ,i A - y ._ ' ,.
  4. 4. ..--. --. --. . . . ..-. ..-.. .. ... ... ... ...-- ,, .---- ..--. , . -- .., ,.- -,.. ... .. . . -- --, -, .-- --, .-.---- ----. .-..4 22:03:12:747 I [INFO] I I I I 0035110/ //dlli/ (dfifb Found Flash uxlder-CSG-Mac-GH,4 22:03:12:747 |[INFO] || I m7/ //A" 'YGVVGVIM 1 Licensestatus 1 ,mapped LEID Flashauilder-CSE-Mac-GH.4 22:o3:12:747 |[mra] |i y/ I 3,/I/44ZXVQ$4;L: :LJU Trial License_4 233313747 I lz V _ f. ._. "`A: Iasn8uilder_4.7 is enabled. .4 22:03:12:747 |_ ""-'-oY: :}, ^8uilder_Base_-$.7 is enabled. .4 22:03:12:747 |"'^4_'-'1lder_Prem_4.7 is enabled. .4 22:03:12:747 I " IVERRIDE_FBP_F0R_TRIAL] in hive I{O1E84E1B-2A54-470B- menenun!in r, ,, ,v ,' -`* , .4 22:03:12:747 I m,_. a - . ..a ,_ ` N ; afval,4 22:a3:12:747 f - - w--"""j:1ble Flashauilder-CSG-Hac-GM.4 22:03:12:747 l;S,;s QHseStatusEx Ended.4 22:03:12:7s I, " "s 3' **Crit duration =permanent,remaining =permanent..4 22:03:12; s" -SQ ST] in user dictionary. DAA.4 2203;;. A _. - ,. ~ gxunointAddrz [a] .4 22:03:13 " - ` ,' l` J SHessaqe Type:[POST] URI: lhttpszl/ lm. licensesmdobe.nents/ vl/ en .4 22:03:14 'uin - 51351] ; Im Code:[B] response size-[179l time taken-[1355.98]ms a f SsnIALs PRESBJT FOR m:USER. .4 4 4 4 22:03:14:114 4 22:03:14:114 4 22:03:1A:114 4 22:03:14:116 4 22:03:14:123 4 22:03:14:124 4 22:03:14:124 .4 22:03:14:124 .4 22:O3:14:124 4 4 4 4 4 4 4 4 .4 12h;Serial data for LEIDS status-Ilma] took-[1356.68)ms. .frori /','V _ 'menentoata call has failed :1130f ,.IM @raiteducensestatusex Started, key [FLHapl in hive [Ftasheuilder-(Ss-Hac-GMI in cache :519t_ gw, , **for key [FLHap] in hive [Ftashsuilder-CSG-Hac-GH(IHZA] in cache :519 : - -' x' z hihihi/ ye for key [Fuiap] in hive [Flashauilder-CSG-Hac-GH(HALL) in cache :519 || ( Xasanurr// / % value for key [FLHap] in hive [Fiashauilder-CSG-Hac-GM(| )LicLocl in cache :51 SLCoreSc-rvjce // /6805 |Query license:type - 3, duration u permanent,remaining - permanent.SLCoreService || |6805 |Ouery license:type a 1, duration - 60 days,remaining a 32 day(s) 3656 second:lI| l| | I 22:O3:14:125 |22:03:14:12S ||| I| II| l i l I22:a3:14:12s || || ODBELib |1 |saas |LEID Found FlashBuilder-CS6-Hac-GH22:03:14:125 || || OOBELiD || |6805 |LicenseType 1 Licensestatus 1 ,mapped LEID Flashauilder-CSG-Hac-GH 22:o3:14:125 || || ooaeun || |saos |Found a valid Trial License22:03:14:125 || || SLCoreService || l 6805 |Feature FlashBuilder_4.7 is enabled. 22:03:14:125 [INFO] || || SLCoreService || 1 6805 |Feature F1ash8ui1der_Base_-1.7 is enabled. 22:03:14:125 [INFO] || || SLCoreservice || |6805 1 Feature F1ash8uilder_Prem_4.7 is enabled. 22:a3:14:125 [DEBUG] || l |PCDService || |6805 |No value for key IOVERRIDLFBILFORJRIAL] in hive HGIEOAOEIB-ZAB-z-UOB-969000189! in master :10
  5. 5. *Vh- G I t 7 7.33%*
  6. 6. ^(? d{4}'d{2}'d{2}) (?d{2}: d{2}: d{2}) (?d+. d+. d+. d+) (?[^s]+) (? [^s]+) (? [^s]+) (?d+) (? [^s]+) (?d+. d+. d+. d+) (?[^s]+) (? d+) (? d+) (?d+) (? d+)$
  7. 7. KNOW Yoni TO0LSITSWMHQRE._THAN JUSCiuster Resource Management. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... C sssssssssssssss Nices
  8. 8. The ApacheTM HadoopCD _project develops open-source software for reliable,scalable,distributed computing. The Apache Hadoop software library is a framework that allows Tor the distributed processing of large data sets acrossclusters of computers using simple programming models.It is designed to scale up from single servers to thousands of machines,each offering local computation and storage.Rather than rely on . hardware to deliver high-availability,the library itself i`s designed to detect and handle failures at' the application layer,so delivering a highly-available service on top of a cluster of computers,each of which may be prone to failures.
  9. 9. Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable,fault-tolerant manner.
  10. 10. The Apache HiveTM data warehouse software facilitates querying and managing large datasets residing in distributed storage.Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL.At the same time this language also allows traditional map/ reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefflcient to express this logic in HiveQL.
  11. 11. Apache SqoopTM is a tool designed for efficiently transferring bulk databetween Apache Hadoopand structured datastores such as relational databases.
  12. 12. Chukwa is an open source data collection system for monitoring large distributed systems.Chukwa is built ontop of the Hadoop Distributed FileSystem (HDFS) and Map/ Reduceframework and inherits Hadoop's scalability and robustness.Chukwa also includes a flexible and powerful toolkit for displaying,monitoring and analyzingresults to make the best use of the collected data.
  13. 13. Flume is a distributed,reliable,and available service for efflciently collecting,aggregating,and moving large amounts oflog data.lt has a simple and flexiblearchitecture based on streaming data flows.lt is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms.lt usesa simple extensible data model thatallows for online analytic application.
  14. 14. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization.lt can be elastically and transparently expanded without downtime.Kafka has a modern cluster-centric design that offers strong durability and fault- tolerance guarantees.
  15. 15. Solr is highly reliable,scalable and fault tolerant,providing distributed indexing,replication and load- balanced querying,automated failover and recovery,centralized configuration and more.Solr powers the search and navigation features of many of the world's largest internet sites.
  16. 16. Morphlines is an open source framework that reduces the time and efforts necessary to build and change Hadoop ETL stream processing applications that extract,transform and load data into Apache Solr,HBase,HDFS,Enterprise Data Warehouses,or Analytic Online Dashboards.
  17. 17. HUE IS A WEB INTERFACE FOR ANALYZING DATA WITH APACHE HADOOP.
  18. 18. -_-- --_. , . _-.., ,._ - -- __, -, .-_ -_, , _ .. .. .. _ .. .. -. SEE iiiiki.. .. .----------- -- .. .. .------- -- || || |008513/ //vj/ l-: EJD Fo; _md Flash uil er-CSG-Hac-GN22:03:12:747 |[INFO] || J %r A? ?? **'30 yoe 1 Licensestatus 1 ,mapped LEID FlashBuilder-CSE-Hac-GH 22:03:12:747 |[INFO] I I / // . jg/ fl/ aexgffZLfziza Trial License22:03:12:747 |[INFO] i /@$04 4 g-; ..$z: ;~__~_. xckrzssnaundegm7 is enabled. 22:03:12:747 |[INFOI/ , f v I . .;, r____, .--. .-. _.. .- ~. -:fe_. :ihauilder_Base_4.7 is enabled. 22:03:12:747 |[INFQ//' _/ / la# ""*; .1lder_Prem_4.7 is enabled. 22:03:12:747 |,/f' *z , #7 f IVERRIDE_FBP_FOR_TRIAL] in hive [[01E04E18-2A64~470B-, . *n ' . l 909000A31n in QSDQA] 22:03:12:747, f 22:03:12:747 I , A22:03:12:74; _ S,Ssnscszatuszx Ended 22:03:12:7$ / _' *x t' (33, duration =permanent,remaining =permanent.22: 3:12;, $ . ' Qs S7] in user dictionary., S 1- duration =60 days,remaining =32 dayls) 3652 secc22:03:12. S ? P OA or expired al '~' ' 'lili l:Flash8uilder-CS6-Hac-Gl4 ]l , key in OPHGetValueForKey X 'on local store in setProxyCredentialsForIALSessLc l 'or IAL session in getEntitlement call _glltizlmenzntexosl*n , x-hhhhbhhknkhlsh33355353353332.. .1 kaents/ vl/ e:A'4 22: :13; ` [Et -4 : :aja _ ._' , nl u! ? 'slim Code:[0] response size-[Us] time taken-[1355.98]ms 4 22:03:14 1;_ _ ,,-&l;l. 4 22: 3:14:11: ' ~ _ ,- , _: . lg f SsiuALs Hassan' FOR m:uscn. .4 22:03:14:114 l _Aerial data for LEIDS status-[1130] took-[1356.68]ms. 4 22:03:14:114 1, letrror4 22:03:14:114 |J. , *$7 lllevnentoata call has failed :11304 22:03:14:116 |V _/"4 22:03::4:123 |z .42'. ? mmliledLicensestatusEx Started4 22:03:14:124 1 . y z-; JWT/ ;ffy/key [FLHap] in hive IFIashBuiIder-CSG-Hac-GH] in cache :519.4 22:03:14:124 |- r, for key [Fuiap] in hive IFlasHBuiIder-CSG-Hac-GH([MZA] in cache :519 .4 22:03:14:124 || |/ ,325.801,75 / / Wild/ ve for key [FLHap] in hive IFlaShBuiIder-CSG-Hac-GM(HALL] in cache :519 4 22:03:14:124 || || I Xxanuav// / MU/ Mr value for key [FLHap] in hive [FlashBuilder-CSG-Hac-GH([}LicLocl in cache :51 4 22:03:14:125 |I l I I SLCoreSe-rvjce // /5805 l Ouery license:type - 3, duration - permanent,remaining - permanent. 4 22:03:14:125 || || |SLCoreService || |6805 |Ouery license:type - I,duration - 60 days,remaining - 32 dayls) 3650 second:4 22:03:14:125 || || |OOBELib || |6005 |Lem Found Flashauilder-CSG-Nac-GH4 22:03:14:125 || || |OOBELiIJ || l 6805 |LicenseType 1 Licensestatus 1 ,mapped LEID FlashBuilder-CSG-Hac-GH4 22:03:14:125 || || |OOBELib || |6805 |Found a valid Trial License4 22:03:14:125 || || |SLCoreService || |6805 |Feature Flash8uilder_4.7 is enabled. 4 22:03:14:125 |[INFO] || || SLCoreService || |6805 |Feature Flash8ui1der_8ase_4.7 is enabled. 4 22:03:14:125 |[INFO] || || SLCoreService || |6805 |Feature F1asn8uilder_Prem_4.7 is enabled. .4 22:03:14:12s |locsucl || || PCOService || |6005 |No value for key [OVERRIDE_FOP_FOR_TRIAL] in hive [(01E84E18-2A64-470B- I2909000A31H in master :10
  19. 19. TPJAT BIG DATA f!THE) ii U W][Jiijli ~J
  20. 20. ANALYSIS CHAINTARGET ARCHITECTURE
  21. 21. g . i9,, _.. ._mm..m mwMm MNM S. ; .a .n . S. ia

Search related