The Cloud's Cutting Edge: ArcGIS for Server Use Cases for Amazon Web

Services David Cordes David McGuire

Jim Herries Sridhar Karra

Jim Herries

Atlas Maps

Atlas sample application

• The Esri Thematic Atlas Sample App is a configurable web application that uses a collection of intelligent web maps with text, graphics, and images to talk about our world.

Lots of maps needed, quickly

Key requirements

• Supply dozens of maps to an online atlas application • Stand up dozens of map services in two weeks • Enable team of map publishers • Option to iterate quickly on cartography, data, popups

CloudBuilder delivered value to the team instantly

• Data lead – knows the data but not CloudBuilder

• Cartographers on team knew color and cartography but not the data and not CloudBuilder

• We all knew ArcGIS server and publishing workflows

• None of us wanted to manage a server in a production situation

CB

743 items 16 apps

Lessons Learned

• Same day startup of server was a huge factor in project success • Teaching the team publishing methods was easy • Cache local, copy up FTP, now tile packages • Hundreds of cached services for the U.S. to 72k scale • 6 hours to cache the US down to 72 k • Use a checklist to save hundreds of hours’ time • Whatever data you have, you’re going to want additional attributes • Your corporate IT network staff are your friends (DNS changes, ports, etc) • Serve dynamic until popups and carto finalized • It’s just another server… • …but we don’t have to worry about it

An ongoing resource • http://atlasmaps.esri.com/arcgis/rest/

David McGuire

Going viral

Cloud Formation Demo

Crisis Thursday Saturday Friday

Servers Down

Traditional Trouble Shooting

• Reboot Servers? • Spread Services Across Local Machines? • Order New Equipment?

Panic!

Transition to the Cloud

Deployment

1 hr 15 mins

30 mins

15 mins

To Stage

To Generate An AMI

To Deploy Live Servers

ELB

ArcGIS Servers

Enterprise Geodatabase

Server

Handle Traffic

What We Learned

Start in the Cloud

Design for the Cloud

Plan Ahead

Avoid internal, static machines

Architecture should be cloud focused

Avoid emergencies

Aggregated Live Feeds…

Frequently Changing Data

How Can I Deploy and Scale?

Cloud Servers

Sync

Sync

Internal Processing

Server

S3 Bucket

Amazon

ELB

ArcGIS Servers

ELB

Web Application

Servers

Hurricane PIM

Hurricane PIM

• Over 1 Million hits in 1 day on our web map • Over 7 Million dynamic map requests in 1 day

Next Phase

Linux

Auto Scaling

Cloud Front

Linux instances are cheaper and boot faster

Avoid manual load management

Take advantage of global content distribution

Complication

Internal Machine

TMAppsEventsS3 S3 Bucket

Production Security Group

US East Region

Availability Zone B Availability Zone C

m1.small proxy server

m1.small proxy server

Internet

Elastic Load Balancing

Amazon CloudFront Distribution

Auto Scaling

VPC Subnet 1 VPC Subnet 2

Resources

• Esri Sample Templates - http://esriurl.com/EsriAWSTemplate

• Amazon Sample Templates - http://esriurl.com/AWSTemplate

• Aggregated Live Feeds - http://esriurl.com/LiveFeeds

- Make sure the “Show ArcGIS Desktop Content” box is checked

Sridhar Karra

Managed Services

Agenda

• Introduction • Program Overview • Customer Deployments

- Architecture - Lessons

• Questions

Program Overview

Extending the ArcGIS platform…

Esri Managed Services Cloud based GIS infrastructure support

REPORTING ARCHIVE

REDUNDANCY PERFORMANCE TESTING

HIGH AVAILABILITY

BANDWIDTH

MONITORING

HARDWARE CHANGE MANAGEMENT

DEPLOYMENT SECURITY

DATA MANAGEMENT

INFRASTRUCTURE

SYSTEM DESIGN SOFTWARE

BACKUP

SCALABILITY STORAGE

DISASTER RECOVERY

CACHING AGOL INTEGRATION

Experienced, Secure, Reliable, Scalable

• Rapid Deployment • Reduced cost of ownership

• Access to Enterprise GIS Expertise • Scalable Resources

• Platforms: - Public Cloud: AWS, Azure, Verizon, IBM - Private Cloud: Esri, Customer Private Cloud, CGI

• FedRAMP / FISMA Moderate Security Offering • ArcGIS Desktop in the cloud support

Road Ahead

Esri Managed Services Cloud based GIS infrastructure support

Esri Managed Services Cloud based GIS infrastructure support

Requirements Proposal / ROM

Staging

Contract

Verification

Questionnaire On Call Sizing / Arch. Design

Production Verification

Prod. Images

Monitoring Alerts / Notifications

Esri Managed Services Cloud based GIS infrastructure support

Requirements Proposal / ROM Contract Questionnaire On Call Sizing / Arch. Design

User Load

Application

Users Count; Peak (Yes / No)

GIS Services (Dynamic, Image, GP, Feature, Cached)

Security

Availability

SSL / AGS Tokens / Data at Rest / Encryption

SLA – 95%, 99% or 99.9%; Monitoring (24/7)

Infrastructure Public Cloud or Private Cloud; DB/Web/App Servers

Esri Managed Services Cloud based GIS infrastructure support

Requirements Proposal / ROM Contract Questionnaire On Call Sizing / Arch. Design

Benchmark

System Designer

PassMark; SystemCPU (AWS, Azure, Private Cloud)

Models; Designing; Capacity Planning

System Test

System Monitor

GIS Services; Web Applications; Service Times

System; GIS; DB; Process; Alerts / Notifications

MXDPerfStat Fix MXD issues (if any)

• http://www.arcgis.com • owner: EnterpriseImp

Esri Managed Services Cloud based GIS infrastructure support

Staging Verification Prod. Images

Deployment

Test

Update

MXD’s (MXDPerfStat); GIS Services (System Test)

Create Snapshot

Esri Managed Services Cloud based GIS infrastructure support

Production Verification Monitoring Alerts / Notifications

Snapshot

Prod. Servers

System

Application (URL)

DB

Process

Emails

24/7 Support

Many successful deployments…

Customer Deployments

Oregon Imagery Explorer

• Cached and dynamic image services • JS Application with Clip/Zip/Ship • SLA – 95%

• Search, download, use large imagery datasets • Simple, easy to use web viewer

Oregon Imagery Explorer (Lessons)

• Deployed in Amazon Web Services (AWS) • 40+ GIS Services (Image, Cache, GP, etc)

- (Min, Max) SOC instances vs. RAM - Paging Files / Swap space

• Raster/Cache data: 22 TB (Approx.) - Ship drives to AWS using Import / Export - Backup before shipping drives (New!!)

• Server Specs: 1 x Standard Extra Large Server - 4 Cores and 15 GB RAM - EBS Volumes: 23 drives

- Max EBS Volume is: 1 TB ONLY

- Windows Stripe (Server OS / Software RAID)

• CloudWatch (Monitor Disk Usage, Custom Counters (service stop, system reboot), CPU Alerts/Notifications)

Iberdrola USA Outage Viewer

• JS Application • Server Auto-Scaling • Data Update Automation

• High Availability • SLA 99.9% • Geographic Redundancy

Iberdrola USA Architecture and Workflow

Load Balancer Load Balancer

Outage Database

Amazon EC2 West

Amazon S3

Amazon EC2 East Primary / Auto-Scaled Cold Backup (DR Site)

Updates Every 15 Minutes

Iberdrola USA Outage Viewer (Lessons)

• Deployed in Amazon Web Services (AWS) • Points to Remember when Auto-Scaling:

- Plan for the Worst - Pick the “right” Instance Type when AS - “Launch” script - Contractual Cap: Max limit for Auto-Scaling - Time needed to launch a new “fully functional” instance - Termination Policy: Yes or No?

- IIS Logs – Usage logs?

- Licensing ? - Check with your Account Manager.

• Alerts and Notifications - System Counters - Data Update Scripts (ArcPy) - CloudWatch (Monitor Disk Usage, Custom Counters (service stop, system reboot), CPU

Alerts/Notifications)

Hurricane Sandy (Usage Statistics – New York)

Hurricane Sandy (Usage Statistics – New York)

Hurricane Sandy (Usage Statistics – Maine)

Hurricane Sandy (Usage Statistics – Maine)

Hurricane Sandy – Outages (New York)

2/10/2014 -11:30 am

Peak Sandy Hours

Heat Map – Fun !!

Maine – October 29

Maine – October 30

Maine – October 31

Maine – November 1

Maine – November 2

Survey

• http://www.esri.com/events/devsummit/session-rater • Enter: “Extending Services in ArcGIS for Server” • 10 seconds • Comments really welcome

FYI: Slides for this presentation are available at: http://1drv.ms/1fnkpTK