View
221
Download
1
Category
Preview:
Citation preview
Mark Fischer
• 20 Years of Web Application Development
• 5 Years of Infrastructure Tools Development
• 2 Years AWS Cloud Automation Development
Overview
• Codify Infrastructure Decisions
• Document Deployment Processes
• Ensure Repeatable Operations
• Empower Developers and Product Owners
how are we going to do user training in prod?
we need another dev environment
*appologies to piecomic.com
Automation Progression
Manual Infrastructure Provisioning
➡ CloudFormation
Manual Environment Configuration ➡ Docker
Manual Code Deployment ➡ Jenkins
Infrastructure Provisioning
A few years ago Operations Staff 1 Week
Last year Better Operations Proceedures 1 Day
Now DevOps & AWS 10 Minutes
Time for me to get new infrastructure provisioned
Security Group Security GroupSecurity Group
DB Subnet Group
X
DB Option Group
10+ Separate Resources
CloudFormation
"AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion."
https://aws.amazon.com/cloudformation/
CloudFormation
• JSON Text Document
• Defines AWS Resources
• Defines Resource Relationships
• Input Parameters for Flexibility
• Provisioning and De-Provisioning
Now With
100%
More Y
AML!
CloudFormation
• Originally All JSON Text Files
• Recently Added YAML Support
"Resources": { "VpcEcsEas": { "Type": "AWS::EC2::VPC", "Properties": { "CidrBlock" : { "Ref": "VPCcidr" }, "EnableDnsSupport": true, "EnableDnsHostnames": true, "Tags" : [ { "Key": "Name", "Value": { "Ref": "VPCName" } } ] } }, "InternetGateway": { "Type": "AWS::EC2::InternetGateway", "Properties": { "Tags" : [ { "Key": "Name", "Value": { "Fn::Join": [ "", [ { "Ref": "VPCName" }, " Internet Gateway" ] ] } } ] } }, "InternetGatewayAttachment": { "Type": "AWS::EC2::VPCGatewayAttachment", "Properties": { "InternetGatewayId": { "Ref": "InternetGateway" }, "VpcId": { "Ref": "VpcEcsEas" } } },
• Parameters (Input Variables)
• Metadata
• Mappings
• Conditions
• Resources
• Outputs
--- AWSTemplateFormatVersion: "2010-09-09"
Parameters: # Pick Zone-A or Zone-B where this EC2 instance will be deployed. AZChoice: Description: "Availability Zone" Type: String AllowedValues: - "Zone-A" - "Zone-B"
Mappings: # The two availability zones where this EC2 instance can be deployed in. ZoneMap: Zone-A: subnet: "subnet-e1c2f584" zone: "us-west-2a" Zone-B: subnet: "subnet-f28fda85" zone: "us-west-2b"
Resources: # Deploys an EC2 instance with some tags. Ec2Instance: Type: "AWS::EC2::Instance" Properties: ImageId: !FindInMap ["OSImageMap", !Ref "OSType", "64"] KeyName: !Ref "KeyName" InstanceType: !Ref "InstanceType" AvailabilityZone: !FindInMap ["ZoneMap", !Ref "AZChoice", "zone"] NetworkInterfaces: - AssociatePublicIpAddress: "true" DeviceIndex: "0" GroupSet: - !Ref "InstanceSecurityGroup" SubnetId: !FindInMap ["ZoneMap", !Ref "AZChoice", "subnet"] Tags: - Key: "Name" Value: !Ref "HostName"
Outputs: InstancePublicIP: Description: "The Public IP address of the instance" Value: !GetAtt Ec2Instance.PublicIp
TemplateAnatomy
# #### EC2 Instance # # Deploys the EC2 instance with some tags. Ec2Instance: Type: "AWS::EC2::Instance" Properties: ImageId: !FindInMap ["OSImageMap", !Ref "OSType", "64"] KeyName: !Ref "KeyName" InstanceType: !Ref "InstanceType" AvailabilityZone: !FindInMap ["ZoneMap", !Ref "AZChoice", "zone"] NetworkInterfaces: - AssociatePublicIpAddress: "true" DeviceIndex: "0" GroupSet: - !Ref "InstanceSecurityGroup" SubnetId: !FindInMap ["ZoneMap", !Ref "AZChoice", "subnet"] Tags: - Key: "Name" Value: !Ref "HostName"
# #### Instance Security Group # # Security group for the EC2 instance, that allows you to SSH into the instance InstanceSecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: GroupDescription: "Allow ssh to client host" VpcId: !Ref "VPCID" SecurityGroupIngress: - IpProtocol: "tcp" FromPort: "22" ToPort: "22" CidrIp: "0.0.0.0/0" Tags: - Key: "Name" Value: !Sub "${HostName} Security Group"
# #### Instance Role # # This is the IAM role that will be applied to the EC2 Instance. Any AWS specific # permissions that the node might need should be defined here. # EnvInstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service:
• AWS Resources are codified in the Template
• Relationships Established
TemplateAnatomy
# #### EC2 Instance # # Deploys the EC2 instance with some tags. Ec2Instance: Type: "AWS::EC2::Instance" Properties: ImageId: !FindInMap ["OSImageMap", !Ref "OSType", "64"] KeyName: !Ref "KeyName" InstanceType: !Ref "InstanceType" AvailabilityZone: !FindInMap ["ZoneMap", !Ref "AZChoice", "zone"] NetworkInterfaces: - AssociatePublicIpAddress: "true" DeviceIndex: "0" GroupSet: - !Ref "InstanceSecurityGroup" SubnetId: !FindInMap ["ZoneMap", !Ref "AZChoice", "subnet"] Tags: - Key: "Name" Value: !Ref "HostName"
# #### Instance Security Group # # Security group for the EC2 instance, that allows you to SSH into the instance InstanceSecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: GroupDescription: "Allow ssh to client host" VpcId: !Ref "VPCID" SecurityGroupIngress: - IpProtocol: "tcp" FromPort: "22" ToPort: "22" CidrIp: "0.0.0.0/0" Tags: - Key: "Name" Value: !Sub "${HostName} Security Group"
# #### Instance Role # # This is the IAM role that will be applied to the EC2 Instance. Any AWS specific # permissions that the node might need should be defined here. # EnvInstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service:
# #### EC2 Instance # # Deploys the EC2 instance with some tags. Ec2Instance: Type: "AWS::EC2::Instance" Properties: ImageId: !FindInMap ["OSImageMap", !Ref "OSType", "64"] KeyName: !Ref "KeyName" InstanceType: !Ref "InstanceType" AvailabilityZone: !FindInMap ["ZoneMap", !Ref "AZChoice", "zone"] NetworkInterfaces: - AssociatePublicIpAddress: "true" DeviceIndex: "0" GroupSet: - !Ref "InstanceSecurityGroup" SubnetId: !FindInMap ["ZoneMap", !Ref "AZChoice", "subnet"] Tags: - Key: "Name" Value: !Ref "HostName"
# #### Instance Security Group # # Security group for the EC2 instance, that allows you to SSH into the instance InstanceSecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: GroupDescription: "Allow ssh to client host" VpcId: !Ref "VPCID" SecurityGroupIngress: - IpProtocol: "tcp" FromPort: "22" ToPort: "22" CidrIp: "0.0.0.0/0" Tags: - Key: "Name" Value: !Sub "${HostName} Security Group"
# #### Instance Role # # This is the IAM role that will be applied to the EC2 Instance. Any AWS specific # permissions that the node might need should be defined here. # EnvInstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service:
• Reference resources created within the template or passed in via parameters
• Order doesn't matter. CloudFormation builds its own dependency graph
Command Line Deployments
• Filling out complex CloudFormation forms is still tedious
• Can create CloudFormation deployments via the aws-cli tools
• Parameters are fed in via a JSON parameters file
Command Line Deployments[ { "ParameterKey": "HostName", "ParameterValue": "fischerm-ec2-demo" }, { "ParameterKey": "KeyName", "ParameterValue": "FischermUAPilots" }, { "ParameterKey": "VPCID", "ParameterValue": "vpc-12a98977" }, { "ParameterKey": "AZChoice", "ParameterValue": "Zone-A" }, { "ParameterKey": "InstanceType", "ParameterValue": "t2.micro" }, { "ParameterKey": "OSType", "ParameterValue": "Amazon-Linux" } ]
--- # EC2 Basic CloudFormation Deployment # ----------------------------------------- # # This CloudFormation template will deploy a single EC2 instance with # its own security group.
AWSTemplateFormatVersion: "2010-09-09"
# Parameters # ---------- # # These are the input parameters for this template. All of these parameters # must be supplied for this template to be deployed. Parameters: # HostName to be used in tagging the EC2 instance. HostName: Type: String Description: "Enter the name of the host or service, ie 'Civil Engineering Structures App', or 'UITS Cloud Services Testing', etc."
# SSH Key Pair to be used on the application EC2 instances for emergency administrative access. KeyName: Description: "Amazon EC2 Key Pair" Type: "AWS::EC2::KeyPair::KeyName" # VPCID is the ID of the VPC where this template will be deployed. VPCID: Description: "Target VPC" Type: "AWS::EC2::VPC::Id" AllowedValues: - "vpc-12a98977"
# Pick Zone-A or Zone-B where this EC2 instance will be deployed. AZChoice: Description: "Availability Zone" Type: String AllowedValues: - "Zone-A" - "Zone-B"
# Default EC2 Instance Type for Application instances. InstanceType: Description: "EC2 Instance Type" Type: String Default: "t2.micro" AllowedValues:
Configuration As Code
• CloudFormation allows you to codify your infrastructure deployments
• Each template deployment will be identical to previous ones
• Plain text files can be versioned and stored in source control
Configuring new App Server
• Following notes from the last time
• Hopefully I wrote everything down…
FROM php:5.6-apache
# Add application dependencies. RUN apt-get update && apt-get install -y \ freetds-common \ freetds-bin \ freetds-dev \ libapache2-mod-auth-cas \ libcurl4-openssl-dev \ libldap-2.4-2 \ libldap2-dev \ libxml2 \ libxml2-dev \ unixodbc \ vim
# Install Well Behaved Extensions RUN docker-php-ext-install \ bcmath \ curl \ json \ ldap \ mbstring \ mssql \ opcache \ pdo_mysql \ soap
# Copy over our application COPY app/ /var/www/html/
# Run our custom startup script CMD ["startup.sh"]
docker git repository
built docker image
Run Image Anywhere That Supports Docker
• Only need to install Docker on a host, no other dependencies
• Lots of Docker enabled environments
• AWS ECR & Elastic Beanstalk
• Azure
• Linode / Digital Ocean / etc.
docker run -d --name yourapp \ -p 80:80 -h yourapp.example.com \ -e "PHP_db_user=ausername" \ -e "PHP_db_pass=secret" \ yourproj/dockerimage
Jenkins
• Really Fancy cron
• Configure jobs to run on-demand or scheduled
• Control access to jobs by user
• Store secrets encrypted & pass into jobs as they're run
Jenkins
• Lots of built-in functionality
• Check out a git repository
• Build a Java Project
• Run shell scripts
• Integrations with services such as Slack, email, SMS, etc
• Chain jobs together on success or failure
docker projectapp source
Nexus
chef cookbooks
cookbooks.tar.gz
template libraryCloudFormation
TemplateCloudFormation
StackOpsWorks
Stack (Environment)Instance
Deployment ApplicationInstances
Jenkins
Secrets
Jenkins
Jenkins
Jenkins docker repository
docker projectapp source
Nexus
chef cookbooks
cookbooks.tar.gz
template libraryCloudFormation
TemplateCloudFormation
StackOpsWorks
Stack (Environment)Instance
Deployment ApplicationInstances
Jenkins
Secrets
Jenkins
Jenkins
Jenkins docker repository
docker projectapp source
Nexus
chef cookbooks
cookbooks.tar.gz
template libraryCloudFormation
TemplateCloudFormation
StackOpsWorks
Stack (Environment)Instance
Deployment ApplicationInstances
Jenkins
Secrets
Jenkins
Jenkins
Jenkins docker repository
docker projectapp source
Nexus
chef cookbooks
cookbooks.tar.gz
template libraryCloudFormation
TemplateCloudFormation
StackOpsWorks
Stack (Environment)Instance
Deployment ApplicationInstances
Jenkins
Secrets
Jenkins
Jenkins
Jenkins docker repository
docker projectapp source
Nexus
chef cookbooks
cookbooks.tar.gz
template libraryCloudFormation
TemplateCloudFormation
StackOpsWorks
Stack (Environment)Instance
Deployment ApplicationInstances
Jenkins
Secrets
Jenkins
Jenkins
Jenkins docker repository
Jenkins
• Restrict access to Jobs
• Certain people can create jobs
• Certain people can run jobs
• Certain people manage secrets
• Allows you to abstract AWS deployment capabilities
• A single AWS IAM User Credential can be used for multiple Jenkins Jobs
• IAM Credentials never leave Jenkins, stay encrypted
Jenkins
• Examples
• App developers can provision & de-provision new environments
• Business Analysts can perform database refreshes (Load new Prod data to Dev for example)
• DevOps staff can manage Jenkins jobs without needing to setup AWS IAM Credentials for Job runners
Sticking Points
• Automation takes more time up front to get right
• IAM Permissions
• Persistant File Storage
• Try and use RDS / S3 as much as possible
• EFS makes this slightly easier (AWS managed NFS service)
Recommended