Technology Trends and Big Data in 2013-2014

  • View
    2.973

  • Download
    0

Embed Size (px)

DESCRIPTION

This is the slide deck that KMS Technology's CTO delivered at The 16th Offline of CIO Vietnam Group on Thurdsay, 23/05/2013.

Text of Technology Trends and Big Data in 2013-2014

  • 1. 1TECHNOLOGY TRENDS FOR 2013Kaushal Amin, Chief Technology OfficerKMS Technology Atlanta, GA, USA

2. ABOUT KMS2Founded in January 2009 with offices in Atlanta, Dublin,Calif., and Ho Chi Minh City, Vietnam, KMS Technology is aUS Offshore Product Development (OPD) company.We have a 400+ global workforce that provides a variety ofcommercial grade web and software development services tosoftware product and technology-based companies. 3. ABOUT SPEAKER KAUSHAL AMIN32011-NowKMS2006-11LexisNexis2001-06Startups1999-02Intel1993-99McKesson1989-93IBM1985-88Engineering Bachelors inComputer Engineeringfrom University ofMichigan Developed OS CrossAssembler in C forMC6809 Developed WindowsNT based optical filesystem for dealingwith large data files Healthcare MedicalRecords & Imaging Wireless mobile fieldservice software onWindows CE and J2ME Developed PriceOptimization softwarefor retail and hotelindustry Provide technicalleadership andmentoring to KMS USand Vietnam staff Provide C leveltechnology consultingto KMS clients Part of OS/2 Kernelteam Atlanta Police MobilePlatform (Motorola) Delta Flight Planning& Fueling Systems inUnix Intels multimediashowcase website in16 languages and 40+countries One of the early N-tierarchitected WindowsCOM+ web system Online BIG DATAsystem of US criminalrecords, education,and employmenthistory on employees LexisNexis s NoSQLdistributed database 4. WHY SHOULD YOU BE HERE Learn about MAJOR software technology trends affecting ITindustry and businesses Necessary in order to anticipate and respond to ongoingtechnology-driven disruptions Step up. Provoke and harvest disruption. Dont get caught unawareor unprepared.4 5. INDUSTRY EXPERTS 2013 LIST5 6. #1 MOBILE APPS6 Mobile devices overtaking PCs as the most commonweb access device worldwide by end of 2013 More market shift towards complex businessapplications instead of small niche consumer apps Similar to PC evolution of desktop productivity apps tonetwork enabled enterprise solutions Apple iOS and Google Android will continue todominate market share for next 2 years Native Apps will continue to be preferred developmentplatform, however, HTML5/Hybrid will start gainingground 7. MOBILE APPS STATS7Mobile App Market Stats: The number of smartphones will exceed 1.82 billion unitsworldwide in 2013 (~ 40% of cell phone market) Android is expected to claim 63.8% market share by 2016 iOS monthly revenues are 4x those of Google Play Apple has paid developers $5 billion in app sales There are now more than 400 million accounts withregistered credit cards in the App Store Google Play Has 700,000 Apps, Tying Apples App Store 8. #2 - BIG DATA8Big data exceeds the reach of commonly usedhardware environments and software tools tocapture, manage, and process it with in a tolerableelapsed time for its user population. - TeradataMagazine article, 2011Big data refers to data sets whose size is beyond theability of typical database software tools to capture,store, manage and analyze. - The McKinsey GlobalInstitute, 2011Volume and Variety of Data that is difficult to manageusing traditional data management technology 9. WHAT IS GENERATING BIG DATA?Homeland SecurityReal Time SearchSocialeCommerceUser Tracking &EngagementFinancial Services9 10. HOW MUCH DATA? 7 billion people Google processes 100 PB/day; 3 million servers Facebook has 300 PB + 500 TB/day; 35% of worldsphotos YouTube 1000 PB video storage; 4 billion views/day Twitter processes124 billion tweets/year SMS messages 6.1T per year US Cell Calls 2.2T minutes per year US Credit cards - 1.4B Cards; 20B transactions/year10 11. TYPE OF DATA Structured Data (Transactions) Text Data (Web Content) Semi-structured Data (XML) Unstructured Data Social Network, SMS, Audio, Video Streaming Data You can only scan the data once as it travels on network11 12. RDBMS LIMITATIONS Very difficult to scale horizontally (more boxes) as thebest way to scale is vertically by utilizing bigger box Physical limited to CPUs, Disk storage, and memory Large servers are too expensive and still cant scale Requires structure of tables with rows and columns Does not deal well with unstructured data Relationships have to be pre-defined through schema Difficult to add newly discovered data quickly12 13. NOSQL CHARACTERISTICS Cheap, easy to implement (open source) Cluster of cheap commodity servers with cheap storage Data are replicated to multiple nodes (thereforeidentical and fault-tolerant) and can be partitioned Down nodes can easily be replaced while cluster is operational No single point of failure Easy to distribute Dont require a schema Massive Scalability Relaxed the data consistency requirement (CAP) less locking and resource contengency13 14. NOSQL SEVERAL OPTIONS Currently 150 implementations and growing(http://nosql-database.org/) Multiple Types based on storage architecture Key-Value Document Column Family Graph14 15. 15EXAMPLE: HEALTHCARE BIG DATAA health care consultancy has made the data coming out of medical practicesthe focus of its thriving business. The company collects billing and diagnosticcode data from 10,000 doctors on a daily, weekly and monthly basis to createa virtual clinical integration model. The consulting company analyzes the datato help the groups understand how well they are meeting the FTC guidelinesfor negotiating with health plans and whether they qualify for enhancedreimbursement based on offering a more cost-effective standard of care.It also sends them automated information to better take care of patients, likecreating an automated outbound calling system for pediatric patients whowerent up to date on their vaccinations. 16. 16EXAMPLE: RETAIL BIG DATAWalmart handles more than 1 million customer transactions every hour,which is imported into databases estimated to contain more than 2.5petabytes * of data the equivalent of 167 times the informationcontained in all the books in the US Library of Congress. 17. 17EXAMPLE: UTILITY BIG DATAWith a smart meter, a utility company goes from collecting one data pointa month per customer (using a meter reader in a truck or car) to receiving3,000 data points for each customer each month, while smart meterssend usage information up to four times an hour.One small Midwestern utility is using smart meter data to structureconservation programs that analyze existing usage to forecast future use,price usage based on demand and share that information with customerswho might decide to forestall doing that load of wash until they can payfor it at the nonpeak price. 18. #3 - CLOUD COMPUTING18 Shift from Should we use to how can we usecloud within corporate IT Personal Cloud to replace PCs for personal contentstorage allowing access across multiple devices Cloud-based disaster-recovery as-a-service De-duplicating and Encryption of data before it is sentto a cloud storage service will be an integralcomponent 19. CLOUD COMPUTING19 Start addressing the real drawbacks of cloudcomputing - the challenges of scale, complexity andchange management - rather than fixating on itssupposed drawbacks such as security, compliance andSLAs Significant growth in SaaS applications in CloudComputing platform 20. #4 - IN-MEMORY COMPUTING20Enabling users to develop applications that runadvanced queries or perform complex transactions,on very large datasets, at least one order of magnitudefaster and in a more scalable way than when usingconventional architectures- Gartner definitionExamples: Fraud Detection Price Optimization Demand Forecast Flight Control Fueling, Maintenance, & Scheduling Simulation (What-If Analysis) 21. IN-MEMORY COMPUTING21Why Now? 64-bit processors allowing access to 16 exabytes ofmemory (32-bit limited it to 4GB) Memory chips getting faster, more capacity, andcheaper due to Moores law New off-the-shelf commodity servers are capable of1TB RAM capacity big enough for many largedatabases to remain in memory In-Memory RDBMS from Oracle, Microsoft, and othersallowing traditional SQL based applications to benefitimmediately by placing data in memory New development tools making it easier for developersto build applications running across multiple bladeservers e.g. 1000 servers 4 cores per server with 512 GB RAM 22. IN-MEMORY COMPUTING22 In-Memory Computing can squeeze batch processesnormally lasting hours into minutes or seconds. These processes are provided in the form of real-timeor near real-time services and delivered to users in theform of cloud services. Numerous vendors will deliver in-memory solutionsover the next two years, driving this approach intomainstream use. 23. #5 - ACTIONABLE ANALYTICS23 To make analytics more actionable and pervasivelydeployed, BI and analytics professionals must makeanalytics more invisible and transparent to their users Embedding analytic at the point of decision or action Real-time operational intelligence systems thatmake supervisors and operations staff more effective Provides simulation, prediction, optimization and otheranalytics, to empower even more decision flexibility atthe time and place of every business process action Enabled by Big Data and In-Memory Computingtechnologies 24. ACTIONABLE ANALYTICS24Examples: Improving Quality of Healthcare by allowing Physiciansto make decisions based on analysis of lab resultshistory, weight, blood pressure, heart rate monitoringfeeds Leveraging CRM data at the point of sell (Amazon) tomake smarter and better decisions Gaining Operational Efficiency via real-time view intodata, processes, and employee productivity Field Service Order Processing 25. #6 SOCIAL MEDIA25 Social Media trend continues to grow and morebusiness applications will leverage social mediathrough integrations The three most trusted forms of advertising are: Recommendations from people I know - 90% Consumer opinions posted online - 70% Branded websites - 70% Mobile in the middle and primary device for use ofsocial media Google+ I