The Cloud: Not Just a Bunch of Fluff Adam J. CooperSoTec 2014 My Background 15+ Years IT & Web Technology Using CDN, SaaS, Cloud for 8+ years Work experience includes Dell, Fox, Intel, McKinsey, Merrill Lynch, Microsoft MBA and Masters Computer Science Talk not representing current or former employers Overview Demystify The Cloud Your personal data in the cloud Survey of some cloud concepts Gotchas Is the cloud right for you? Explore Amazons AWS The Cloud is Not Magic The Cloud: servers you interact with that are owned or managed by someone else and are often located remotely Multi-Tenancy: often you rent space and share that space with others Meet The Cloud The Cloud is the Internet The cloud is a metaphor for the Internet. Its a rebranding of the Internet By virtue of being a metaphor, its open to different interpretations. Reuven Cohen, co-founder of Cloud Camp Source: MIT Technology Review 2011 (http://www.technologyreview.com/news/425970/who-coined-cloud-computing/)http://www.technologyreview.com/news/425970/who-coined-cloud-computing/ The Cloud is a Branding Tool [And] its worth money. Reuven Cohen, co-founder of Cloud Camp Common Cloud Uses Desktop or Server Virtualization Storage Software applications or platforms File or content sharing or social media Failover or redundancy (DR or failover or geo-location) Hosted databases (including NoSQL) Improve on the Internet (DNS routing, CDN acceleration or caching) Big data storage and analysis Bursty or seasonal loads Media sharing, processing, and streaming Static Web hosting Primary data centers (best for small businesses) Origins of the Cloud 1996: Compaq Computer marketing team envision all business software server from Web as cloud computing- enabled applications May 1997: NetCentric (now out of business) submits trademark application for educational services running on cloud computing August 2006: there is an emerging new model. It starts with the premise that the data services and architecture shold be on servers. We call in cloud computing Eric Schmidt, Google CEO at Search Engine Strategies Conference 2007: Amazon, Microsoft, IBM start offering cloud-computing services Source: MIT Technology Review 2011 (http://www.technologyreview.com/news/425970/who-coined-cloud-computing/)http://www.technologyreview.com/news/425970/who-coined-cloud-computing/ Well Known Cloud Vendors Who Uses The Cloud? Everyone: File sharing Social media Your data stored in the cloud: Online transactions history (Amazon) Brick and mortar loyalty cards (grocery stores) Other personal data Major Companies, Governments, and Universities Your Data in the Cloud Ditto Labs scans public social media photos for marketing campaigns Gmail scansand remind you when bills are due Green Button review electricity usage myAT&T update TV recordings MyUCSDChart doctors appointments, lab results, bills Web tracking Your Data for Ad Targeting WSJ Article Yesterday (10/24) Title: Why You Cant Trust Youre Getting the Best Deal Online A Study Finds Discriminatory Pricing on E- Commerce Sites Is More Widespread Than Thought The Web is full of personalized content, whether its a Netflix recommendation or the results of a Google search. But consumers have protested when e-commerce companies have extended their behind-the-scenes personalization to prices, charging different sums for the same goods 1000 Genomes Project "At 200 terabytes - the equivalent of 16 million file cabinets filled with text, or more than 30,000 standard DVDs - the current 1000 Genomes Project records are a prime example of big data that has become so massive that few researchers have the computing power to use them." "Providing cloud access will expand the universe of researchers who have access to the data..." "This timely initiative will generate tools and approaches for maximizing the return on our national investments in large- scale data collection." "Now we want to find new and better ways to make the most of these data to speed discovery, innovation and improvements in the nation's health and economy." - NIH News, HHS https://www.genome.gov/ https://www.genome.gov/ "Big Data" Wikipedia: "Big data is an all-encompassing term for any collection of data sets so large and complex that it becomes difficult to process using traditional data processing applications." Often requires storage and processing from many, many servers Cheap storage and ability to scale up short-term make The Cloud a perfect candidate for storing and processing this data Cloud Types Software as a Service (SaaS) Google Docs, SalesForce, Office 365 Infrastructure as a Service (IaaS) Most of AWS and Azure Platform as a Service (PaaS) Box, OneDrive, Office 365, Zocalo Other: Social media Desktop virtualization (AWS WorkSpaces) Content Delivery Networks (CDN) Focus on IaaP Cloud Pros Lower fixed costs Cheap to experiment Some services lend themselves well Reduce server management overhead Reduce application or server setup times Reduces physical desktop requirements Benefit from shared scale / features provided by cloud vendor (e.g. security) Developer flexibility Cloud Cons Higher variable costs Vendor stickiness Security concerns (real or perceived) Regulatory concerns Requires special expertise to harden infrastructure Susceptible to vendor outages Higher application bandwidth requirements Governance may be harder Everything is do it yourself Lack of tooling (currently) Cloud Gotchas Forgetting to spin down unused instances Not architecting for security Not architecting for redundancy Publishing apps with root admin keys Disregarding least privilege concepts Ignoring monitors Not training existing staff or hiring specialized architects Ignoring savings opportunities (e.g. reserved instances) Disregarding elastic load balancing Thinking clouds dont go down Forgetting bandwidth constraints A Few Recent Outages 12/25/2012 AWS outage brings down Netflix and other services (on Christmas) 8/25/2013 AWS outage brings down Instagram, Vine, and others 2/20/14 Google Docs / Drive outage 8/2014 Azure outages 6 times in 9 days affecting various services globally Don't Be the Next CodeSpaces 2 Mistakes: Lost their keys Failed to backup Top Cloud Threats NoThreat 1Nefarious use 2Insecure APIs 3Malicious insiders 4Shared technology 5Data loss or leakage 6Account hijacking 7Unknown reason 8Hardware failure 9Natural disaster 10Vendor closure 11Cloud-related malware 12Inadequate design Source next three slides: Cloud Securities Alliance 2013 (https://www.scribd.com/doc/ /Cloud-Computing-Vulnerability- Incidents-A-Statistical-Overview)https://www.scribd.com/doc/ /Cloud-Computing-Vulnerability- Incidents-A-Statistical-Overview Frequency of Cloud Threats Threats By Provider Cloud Governance Spin down under-utilized capacity Monitoring Tagging Least privileges (user and apps) Segregation of duties Security account management Firewalls Multi factor authentication (MFA) Evangalize Pros Dont forget the cons Architecture review committee Multi Factor Authentication Step 1: What do you know Step 2: What do you have Comparing Deployment Models On-Premise Cloud Hybrid Consider: Cost (including nature of application) Skillsets Regulatory requirements / security concerns Performance and reliability needs Cloud Considerations Costs and type of load Feature requirements Performance and reliability Legal issues Skillsets Security requirements and certifications Governance Vendor support and forums Staffing for the Cloud Architects are crucial (specialized to cloud) Network engineers remains essential (traditional skills) Developer / QA the same A few new cloud SDKs May use existing platform knowledge (LAMP, Windows, etc.) Infrastructure team has smaller role on the cloud Still need monitoring, patching, security, and deployment Getting Started Read and do trainings Try (learning on the cloud is cheap) But shut down when done And dont use production instances Take advantage of AWS free tier Is the cloud right for you? Determine your strategy (hybrid / all-in) Staff appropriately Create governance early Tools (for security, bill back, reliability, etc.) Exploring AWS Gartner Magic Quadrant Gartner Report Quotes [AWS] is a thought leader; it is extraordinarily innovative, exceptionally agile, and very responsive to the market. It has the richest array of IaaS features and PaaS-like capabilities, and continues to rapidly expand its service offerings. "[AWS] is the overwhelming market share leader, with more than five times the cloud IaaS compute capacity in use than the aggregate total of the other 14 providers in this Magic Quadrant." AWS has a very large technology partner ecosystem. Many software vendors have specially licensed and packaged their software to run on EC2, either independently or via the AWS Marketplace Some AWS Customers AWS History "There's a hidden Amazon, just under the epidermis, the guts of Amazon, this is all the stuff we have to do on the back-end to make this work... [representing] 11 years of web-scale computing [and] billions of dollars in technology and content investment". - Jeff Bezos, 2006 "It struck us in the infrastructure engineering organization that we really needed to decentralize the infrastructure by providing services to development teams" - Jeff Pinkham Amazon Engineer Pinkham proposes concept in 2003 SQS launched publicly in late 2004 EC2 launched publicly in 2006 Sources:awss-ec / and Wikipediahttp://www.zdnet.com/how-amazon-exposed-its-guts-the-history-of- awss-ec / How Much Comes From AWS? Source:Did Azure / Google cause this decline? (total Amazon revenue: $19.3 bln qtrly) AWS IaaS Services Security VM and VPC Storage CDN and DNS DB Analytics and big data Cloud automation and monitoring Other (queue, mobile, search, etc.) AWS IaaP Services (New) Desktop (WorkSpaces) Enterprise storage and sharing (Zocalo) AWS Directory Service (New) Connect AWS to on-premise Microsoft AD Allows more hybrid deployment scenarios Introduced to catchup to Azure which has had this offering for a while AWS Source: Amazon AWS A Few AWS / Cloud Concepts Identity and Access Management (IAM) No charge Public / Private Key Pair Password (optional MFA) Fundamental to all other services Amazon Machine Image (AMI) Regions and Availability Zones Content Delivery Network (CDN) Current Regions and Edges North America and Gov Cloud (N. CA, Oregon, Virgina) South America (Sao Paulo) Europe, Middle East, and Africa Asia Pacific (Beijing, Singapore, Sydney, Tokyo) Source: Amazon AWS AWS Availability Zones Source: Amazon AWS CloudFront CDN Acceleration Source: Amazon AWS CloudFront CDN Caching Source: Amazon AWS Developing AWS Web UI (Desktop and mobile) PowerShell or CLI SDKs REST (Some features require REST or CLI) AWS SDKs and Platforms (e.g. JavaScript) Source: Amazon AWS Getting Started With EC2 Amazon Machine Image (AMI) Elastic Block Storage (EBS) Elastic IP Dont forget Availability Zones Consider AWS Marketplace EC2 AMIs Include Windows Windows with MS SQL Linux (various flavors) Upload your own AMIs Three Purchase Options On Demand Spot Reserved Instance Partial EC2 Configurations Partial Configurations Cont. Security Available for All But Dont Mess It Up! Training AvailableLabs (many free) Instructor training https://forums.aws.amazon.com/index.jspahttps://reinvent.awsevents.com/ AWS Reference Architectures Source for following slides: Nave Web Application AWS Web App Reference Arch. Media Pipeline (SQS) AWS DR Scenario