Building Modern Digital Services on Scalable Private Government Infrastructures using Open Source Technologies for Public Service

Certificate of Good StandingKnowledge Transfer

Presentation

New PR.gov InfrastructureGood Standing Certificate Service

By Andrés Colón Pérez

About me

• Architect for the Good Standing Certificate System• Office of Management and Budget Employee

previously assigned to the Chief Information Officer Office

• Worked with a multi-agency team to identify the problems and solutions

• Designed and developed the PR.gov infrastructure, networks, applications and deployment automation

About this Presentation

• An attempt to familiarize government personnel with:– building modern digital services– using free open source software to reduce

licensing costs– designing networks, building and deploying

applications– designing scalable government systems

What you’ll learn over the weeks:• Project Overview (in spanish) • Overview of concepts and technologies used in design, app development,

and infrastructure, including:– HaProxy– Keepalived– Bind9– Redis– Ruby– Resque– Padrino, Grape, Puma, EventMachine and other gems– HTML5 and Twitter’s Bootstrap– Git– Ansible– PhoneGap

Week 1

Knowledge TransferNew PR.gov Infrastructure

Good Standing Certificate Service

What’s the Plan?

• Overview of Knowledge Transfer Plan• Knowledge Transfer Talk Topics: Week 1

Before we get started:

• Topics serve as an introduction• Feel free to ask questions• You’re expected to read up more on your own• Five minute breaks

KNOWLEDGE TRANSFER PLAN

Knowledge Transfer Plan

• Weekly Presentations– Save your calendar: Fridays 2:30pm– June to September– Starts June 19, end September 11 2015

• Weekly Q&A, Thursdays 9:30am – 10:30am• Multiple Topics

Week 1 Topics

• Overview of concepts and technologies in:– Development– PR.gov Infrastructure / Network Design

Development Topics

• Software Architecture Style (Micro-Services)• Application Programming Interfaces• Redundancy and Scalability• Asynchronous vs Synchronous Services• Open Data (data.pr.gov) • Version Control • Open Source (Github / Application Stack) • Software Development Method (DevOps)

MICRO SERVICES

Monolithic Applications

Monolithic Applications

Monolithic Applications (1990s)

• Tight coupling• Everyone must agree on changes• Each change has unanticipated effects

requiring careful testing beforehand • Harder to Scale

Micro Services (2010s)

• Software Architectural Style• Complex Applications composed of small

proccesses• Loose Coupling• Easier to Scale

Micro Services

APPLICATION PROGRAMMING INTERFACES

API’s for Short

• Set of Protocol, Routines & tools for building software applications

• Goal: interoperability• A Web Service is a type of API• Most popular Web Services:– Simple Object Access Protocol (SOAP)– Representational State Transfer (REST)

• SOAP often referred to as WebService• REST often referred to as REST API

REST APIs

• Versioning is important– /v1/apiname/resource– /v2/apiname/resource

• Must properly implement HTTP error codes• Commonly return JSON, YML.

REDUNDANCY AND SCALABILITY

Handling Load

ASYNCHRONOUS VS SYNCHRONOUS SERVICES

Synchronous

Asynchronous

Open Data

Open Source

Version Control

• System that records changes to files and set of files

• You can easily recall specific versions• Great for collaboration:– Branch– Merge – Revisions

• Popular version control: GIT, Mercurial • GITHUB != GIT

DevOps

DevOps

DevOps

• Short for Development and Operations• As Systems Scale, automation is critical• Do more with less• Consistent and easily deploy servers, manage

networks and applications

Ansible

Ansible

• Server Automation for Humans• Based on SSH• YAML Configuration Files• Doesn’t require dedicated server• Uses Playbooks

PR.gov Infrastructure Topics

• Security Philosophy• Networks Segmentation• Virtual Router Redundancy Protocol• Documentation

SECURITY

Networks in the new Infrastructure

• Why Segmented Networks?• Servicios PR.gov Networks

Why Segment Networks?

• Splitting networks into subnetworks• Boosting Performance• Improving Security

VIRTUAL ROUTER REDUDANCY PROTOCOL

High-availability

PR.gov Networks

• Public Load-Balancing Network– Redirect Public Traffic

• Front-End Network– Process Public Traffic

• Private Network – Inter-agency and intra-services network

• Back-end Network– Private data storage

Next week

• VRRP in Linux• Haproxy Load balancer• SSH Authentication

Week 2



What’s the Plan?

• Last week Q&A• Overview of Knowledge Transfer Plan Week 2• Knowledge Transfer Talk Topics: Week 2

Last week we saw:

• CAP Project Overview• Overview of concepts in:– Development– PR.gov Infrastructure / Network Design

Last Week: Development Topics Q&A

• Software Architecture Style (Micro-Services)• Application Programming Interfaces• Redundancy and Scalability• Asynchronous vs Synchronous Services• Open Data (data.pr.gov) • Version Control • Open Source (Github / Application Stack) • Software Development Method (DevOps)

PR.gov Infrastructure Topics Q&A

• Security Philosophy• Networks Segments• Virtual Router Redundancy Protocol• Documentation

Overview of technologies in Operations:

Infrastructure Services:• Keepalived (implementación de VRRP en Linux)• HaProxy (Load-Balancing en Linux)– Introduction– HTTP Load-Balancing– TCP Load-Balancing– Health Checks

• TLS• Key Authentication

KEEPALIVED

KEEPALIVED

KEEPALIVED

KEEPALIVED

Keepalived:

• Definitions• About• Installation• Configuration• Logging

High-availability

“A system that is continuously operational for a desirably long length of time”

High-availability Goal:

High-availability in Servers:

Keepalived• Free Open Source Software• Zero licensing costs• Written in pure C• Used for High-availability• Implements VRRP• I/O Multiplexer provides realtime networking• Robust and Stable

Keepalived & VRRP• VRRP is IETF protocol• Allows two or more routers to act as a virtual

router• Routers present a Virtual IP Address (VIP) that

corresponds to a Virtual Mac Address (VMAC)• Each router has a real hardware and IP

address

Keepalived & VRRP• Linux does not support Virtual MACs.• Keepalived only implements VIPs, works fine

on all modern networks • Requires a network that allows gratitious

Advanced Resolution Protocol (ARP) requests– An advanced notification– Updates cache ARP cache before other systems

ask for them

Keepalived & VRRP• Backup Server(s) monitor continuously– Listens for multicast advertisements – Expects them from the current master server

• If master disappears– An election process ocurrs– The highest priority backup wins– Winning backup announces gratitious ARP for that

VMAC, and takes over– Happens almost instantly

Keepalived

Keepalived: Software Design

• IP Virtual Server: provides transport-layer load balancing inside the Linux kernel

• Layer-4 switching (OSI)• Allows things Linux Virtual Servers (LVS)– Cluster of servers – Appears as single server to user– Layer 4 balancing

• Note: we dont use LVS

Keepalived & IPVS

OSI Layers

Keepalived Linux Process

Keepalive(d) your Load-Balancers

Keepalived is simple in Linux

• Install: – apt-get install keepalived

• Configure:– vim /etc/keepalived/keepalived.conf

• Start Keepalived:– service keepalived start

• Stop Keepalived:– service keepalived stop

Keepalived Logs

• Read the logs: – tail /var/log/syslog

• How it looks:

Keepalived Configuration

Location:/etc/keepalived/keepalived.conf

global_defs {}vrrp_script chk_service { # Requires keepalived-1.1.13 script "killall -0 keepalived" # cheaper than pidof interval 2 # check every 2 seconds weight 2 # add 2 points of priority if OK}vrrp_instance VI_1 { state master interface eth0 virtual_router_id 52 priority 100 advert_int 1 authentication { auth_type <PASS-TYPE> auth_pass <PASSWORD> } virtual_ipaddress { 192.168.108.10 dev eth0 label eth0:0 } track_script { chk_service }}

HAPROXY

HAProxy: What is it?

• Realible, High Performance Load Balancer• Can Load-balance both TCP and HTTP• Can handle massive amounts of traffic• Can queue up requests for a server• Can be configured to send specific amounts of

traffic to an application:– Configure to never sends more than you can handle– Helpful for heavy apps

HAProxy: Who uses it?

• Who uses it? Internet Giants:– Twitter– Instagram– Reddit– Tumblr– Airbnb– Farmville– Imgur– Github– TaoBao: Largest Picture Content Distribution Network in the World – <insert censored site> uses it

HAProxy:

• How do we use it?– HTTP Load-Balancing– TCP Load-Balancing– Health Checks– Statistics

YOU TOO CAN SCALE APPS

HAProxy is a breeze to setup:

• Install: – add-apt-repository ppa:vbernat/haproxy-1.5– apt-get install haproxy

• Configure:– vim /etc/haproxy/haproxy.cfg

HAProxy, simple administration:

• Start:– service haproxy start

• Stop:– service haproxy stop

• Restart:– service haproxy restart

HAProxy is simple & powerful:

HAProxy s Keepalived

HAProxy is simple & powerful:

TLS

Transport Layer Security

• TLS is a protocol that ensures privacy between communicating applications and their users

• When a server and client communicate, TLS aims to ensure that no third party may eavesdrop or tamper with any message.

• TLS is the successor to the Secure Sockets Layer (SSL).


• SSLv1, SSLv2, SSLv3 and TLSv1 are all vulnerable

• Weakest protocol supported by the system is used for attacks– POODLE– HEARTBLEED


• Check your servers for insecure cyphers and protocols:– https://www.ssllabs.com

• Man in the middle attacks can decrypt HTTPS data

https://www.ssllabs.com/

https://www.ssllabs.com/


SSL Termination

SSL Termination (HAProxy)

SSL Termination

KEY AUTHENTICATION (SSH)

Secure Shell

• SSH is a cryptographic (encrypted) network protocol • Used for initiating text-based shell sessions • Used for administering remote machines in a secure

way– Network Switches– Servers

• Can authenticate using:– Username / Password (less secure)– SSH Keys (more secure)

Secure Shell

• /home/acolon/.ssh/– Same as: ~/.ssh/

• Contains:– Cryptographic keys– Authorized Keys and machines

SSH Hidden Folder

• cd• mkdir ~/.ssh • chmod 700 • cd ~/.ssh/• ssh-keygen -t rsa

SSH: Generate your Keys

SSH: Generate your Keys

• Default is 2048 bit key• Use: ssh-keygen -t rsa -b 4096

SSH: Transfer your Keys

• One way:– ssh-copy-id <username>@<host>– Example: ssh-copy-id [email protected]

• Or:– Copy your public key to the server– cp authorized_keys authorized_keys_Backup– cat id_rsa.pub >> authorized_keys

mailto:[email protected]

Secure Shell

Secure Shell

Why SSH Authentication?

• Far more Secure• Difficult to crack• Less prone to Brute Force attacks• Multiple users can authenticate

Week 3



What’s the Plan?

• Last week Q&A• Overview of Knowledge Transfer Plan Week 3• Knowledge Transfer Talk Topics: Week 3

Q&A - Last week we saw:

Infrastructure Services:• Keepalived (high-availability with VRRP in Linux)• HaProxy (Load-Balancing in Linux)

– Introduction– HTTP Load-Balancing– TCP Load-Balancing– Health Checks

• Transport Layer Security (TLS) & SSL vulnerabilities• Passwordless Authentication – SSH Key Authentication

Overview of technologies in Operations:

Infrastructure Services:• Advanced Key Value Store: Redis• PostFix• Bind9

Databases

• There isn’t a “one-size fits all” • Choosing the right tech, hinges on use case• If your data doesn’t change and has moderate

manageable growth: SQL not dead for you• High throughput and growth, efficient scaling,

rapid data change: NoSQL

RDBMS• Scalability:

– Scability is vertical– More data usually means bigger servers– Scaling across multiple servers is possible but time-consuming

• Fixed Schema– Must be decided and locked before data entry

• ACID compliance• Stored in Relational Model

– Rows: contain all information about specific entity– Columns: contain all the seperate data points about entity.

Choose the Right Tool for the Job

Polyglot Persistence

“Use the right tool for what you’re trying to accomplish”

CAP:Our project uses both SQL and NoSQL– PR.gov successfully implements NoSQL (Redis)– RCI uses both SQL and NoSQL (MSSQL, MongoDB)

What is NoSQL?

• An alternate way of thinking about databases• NoSQL = “Not Only SQL” • Not a Relational Database• Data not modeled in terms of tabular relations• Some NoSQL databases are ACID compliant,

but some sacrifice compliancy for performance and scalability

How many types of NoSQL?

There are plenty of NoSQL flavours:• Key-Value Stores• Document databases• Graph Databases• Wide Column Databases

NoSQL Adoption

• Size Matters:– When working with large datasets, consistently scaling is

easier to achieve with many NoSQL family• Speed:– NoSQL is usually faster and sometimes extremely

speedier in writes– Reads can also be very fast depending on the NoSQL DB

used and data being queried• NoSQL has seen rapid adoption in web-technologies

Why NoSQL?

Data in NoSQL

• Key-Value Stores– Associative Array of key-value Pairs

• Document databases– Stored as collection of document, structure can vary

• Graph Databases– Data is stored in nodes, properties and lines

• Wide Column Databases– Data is stored in column families, rows can have

different columns

Popular NoSQL Examples

• Key-Value Stores– Redis

• Document databases– MongoDB

• Graph Databases– Neo4J

• Wide Column Databases– Cassandra & Hbase

Examples of NoSQL

NoSQL Adoption

IN-MEMORY DATABASEURL: Redis.io (Remote Dictionary Server)

Introduction to Redis• Most important feature: high-performance• Advanced Key-Value Store• Often referred to as a Data Structure Server• Open Source (BSD license)• Built-in replication • Multiple Persistence Options• Read and Write speeds obsessively

documented

Redis: high-performance• In-memory database• Small code-base (20k lines in C)• Connection via TCP or Unix Socket• No nested data structures• Persistence via Snapshotting and/or Journaling• Master/Slave chain database replication• Sentinel Server Monitoring – real clustering

now in beta

Redis: Uses Cases• Caching• Statistics collection (downloads, hits, time

benchmarks)• Log buffers• Tasks Queues• Share state between processes • Inter-proccess communication in a distributed

network• Built-in Publish Subscribe

Who uses it?• Twitter• Instagram• Pinterest• Snapchat• StackOverflow• Airbnb• Tumblr• Flickr• Craiglist• Hulu• Imgur

Redis: Why we love it!• Automatic Key Expiration • Great for both caching and storage• Scales for millions of requests • Used for fast, self-expiring sessions on Web App• Used for fast, self-expiring transactions on GMQ • Powerful Libraries available for EventMachine

(Reactor Pattern) used by our GMQ API for Redis• Redis used by Github’s Resque for asynchronous

workers

Cap Redis• Used primarily as:– Web Session Storage – Transaction Storage– Workers Coordination – Statistics

Redis: Data Structure• Often referred to as a Data Structure Server– Can contain Strings– Hashes– Lists– Sets– Sorted sets– Bitmaps

Redis: Data Types

Redis: Master the Data Types• Redis can be used as Key Value storage• But to get the most out of it, think of it as a tool set• When designing for efficiency, think how to best model

your data using the myriad of available data types• Think of how you want to store your data, including the

key

Redis: InstallationQuick install:

Detailed configuration:http://redis.io/topics/quickstart

Redis: Starting and Shutting down

Start your server:

Shutdown your server:

Redis: Connect to Redis

• redis-cli allows you to connect to a redis server

• It accepts arguments as commands, such as:



Redis: Testing Redis


Redis: Hashes in Redis

Think of hashes as:

• “users:1” => { :name => “andres”, :rank => 1 }


Redis: Connect Ruby to Redis

Redis: Connect Ruby to Redis

Redis: Pub/Sub

Redis: Pub/Sub

Redis: Persistence• In-memory

– No storage on disks. Useful for caches• RDB

– Favors performance over persistence– Very compact single-file representation– Perfect for Backups (backup daily, keep snapshots for months)– Very good for disaster recovery (compact, easily transferable)– RDB maximizes performance since all the parent process needs to do in order to persist is forking a

child that will do all the rest. The parent instance will never perform disk I/O or alike.– Can save every X seconds or if more than Y number of transactions have been changed

• AOF– Favors persistence over performance– Much more durable than RDB– An append-only log, there are no seeks nor corruption problems if power failure– If log ends with half-written command (disk-failure, etc), redis-check-aof tool fixes it easily– Much bigger than RDB– More aggressive storage, as it favors persistence

Redis: Replication

EMAIL SERVERMail relay

Email

• One of the most popular internet services to date

• Facilitates communication• MTA – mail transfer agents, move mail from

one mail system to another• MDA – mail delivery agents, move mail from

one system to the user’s mailbox

Postfix• Free open source mail transfer agent (MTA)• Handles routing and delivery of email• Solid Email Server for Linux• The default MTA for a number linux

distributions including Ubuntu• Very useful for SMTP Relay• Quick setup, very reliable

Postfix Queues• Incoming Queue:– Receives mail from other hosts– As long as emails are arriving and it hasn’t been

processed, it is kept in this queue• Active queue:– The queue that actually deliver messages– It has a limited size and messages are accepted if

there is space for them. Other queues must wait for the active queue to be ready to accept items.

Postfix Queues• Deferred queue:– Email that cannot be delivered– Prevents the system from continously trying to deliver

email– Keeps the active queue short, by storing failed emails,

and thus newer messages get priority– Enhances stability– If MTA cannot reach a domain, emails are stored here– Retry is scheduled with an increasing waiting time. – After wait, the item is put on the active queue.

CAP and email relays

Postfix: how we use it

• GMQ provides a REST interface for mailing• GMQ workers queue jobs in relay server• Postfix mail server is not exposed to the

internet• Traffic is only outbound, not inbound

Postfix: Installation

• sudo apt-get install postfix• Select “Internet Site”• Enter name of your domain• Additional configuration:– Edit: /etc/postfix/main.cf

• Sender Policy Framework (SPF) record is important for the domain you wish to relay

Postfix: Installation

• Postfix start – starts the server• Postfix stop – stop the server• Postfix reload – reloads configuration without

downtime

Postfix: Important commands:

• check queue size:– mailq

• Check current queue:– postqueue –q

• Flush the queue (force resend):– postqueue –f

• Show number of emails being sent to each domain:– qshape active

• Same as above but for deferred queue– qshape deferred

BIND9DNS

DNS

• Allows not to hard-code IPs in our network• Possible to associate multiple names to the

same machine to update the different available services

• Makes our infrastructure more resilient to future changes

Bind9

• Free Open Source DNS Server• Massively popular in the linux community• Resilient and easy to install and configure• Allows for master and slaves DNS• Allows for zone transfers

Commands

• Install:– sudo apt-get install bind9

• Start:– /etc/init.d/bind9 start

• Restart:– /etc/init.d/bind9 restart

• Stop– /etc/init.d/bind9 stop

Configure

• Files:– /etc/bind/

• Local configuration:– /etc/bind/named.conf.local

Configure

• Files:– /etc/bind/

• Local configuration:– /etc/bind/named.conf.local

Configure

• Zones:– /etc/bind/zones/

Configure

• Zones:– /etc/bind/zones/

Next week (4) – Save the Date!

Basic Core Application Technologies:• Ruby Programming Language• Gems• Environment variables (DotEnv)• Rake• Bundler• Internationalization (i18n)

Week 4



What’s the Plan?

• Last week Q&A• Overview of Knowledge Transfer Plan Week 4• Knowledge Transfer Talk Topics: Week

Q&A - Last week we saw:

Infrastructure Services:• Advanced Key Value Store: Redis• PostFix• Bind9

Basic Core Application Technologies:

Infrastructure Services:• Ruby• Gems• DotEnv• Rake• Bundler• Internationalization (i18n)

RUBY

Topics

• Some thoughts on Programming Languages• Ruby History• Who uses Ruby• Ruby Basics• Learn Ruby by Example

The Principles of Languages

• Thinking is Important for Programmers, we can only code what we can think

• But how do we think?– In words of specific languages– We grasp the world by language and express

ourselves with them– Languages are not only tools to communicate but

also tools to Think

Programmer’s Thoughts

• Natural languages are:– too ambiguos,– too verbose– too indirect

• In code, written down thoughts become programs

Ever been frustrated with Programming?

Not all languages are created equal

Everyone can become frustrated with a specific Language. Try a new one.

About Ruby

Ruby is a dynamic, object-oriented, general-purpose programming language.

About Ruby

Ruby is a:• dynamic, • object-oriented, • general-purpose programming

language

Dynamic Languages

High-level programming languages which at runtime, execute many common programming behaviors that static programming languages perform during compilation.

Object Oriented Programming (OOP)

A programming paradigm based on the concept of "objects”, which are data structures that contain data, in the form of fields, often known as attributes; and code, in the form of procedures, often known as methods.

General Purpose Language

In computer software a general-purpose programming language is a programming language designed to be used for writing software in a wide variety of application domains.

About Ruby: History

• Relatively young, 1995• From Japan• Designed to be Natural• Grew hugely in popularity with the Rails

Framework (Ruby on Rails)

About Ruby

• Free:– Developed as open source with a very open

license– Freedom to learn from the source– Freedom to extend and modify

About Ruby

• Ruby is strong in scripting as Perl– Built in regular expressions– Almost all equivalent functionality

• Can access all system calls on the Operating System via a standard library– Ruby/DL (Dynamic Loading)– Explicit libraries: syscall, Win32API

• Useful for scripting, but not limited to it

About Ruby

• Ruby’s OOP Features:– Object• Everything is an object

– Class• Every class is an object

– Methods• Every procedure is a method;

OK! LET’S COMPARE“Hello World!”

Hello World: LotusScript

%INCLUDE "symphonylsx.lss"

Dim application As SymphonyApplication Dim documents As SymphonyDocuments Dim document As SymphonyDocument Dim range As SymphonyTextRange Set application = New SymphonyApplication Set documents = application.Documents Set document = documents.AddDocument("",True,True) Set range = document.content.End Call range.InsertBefore("Hello World")

Hello World: C#

Hello World: Objective-C

Hello World: Visual Basic

Hello World: Java

In Ruby

puts “Hello, World!”

Ruby is focused on programmer productivity

over machine optimization

Ruby Uses

• Simulations• 3D Modeling• Business• Robotics• Networking• Game Development• System Administration• Web Applications• Security

Who is Using it?

• NASA (Langely Research Center)• Google (Sketchup) • Lucent (3G wireless telephony product)• Level 3 Communications (central data collection for

over 1,700 global servers)• 37Signals (Basecamp)• Twitter• AT&T (YellowPages.com) • StateFarm (R&D Center)

Ruby Features

• Cross Platform • Object Oriented• Powerful string operations• Variables are not typed• Regular Expressions

Ruby Features

• Class Inheritance• Garbage Collection• Threads• Iterators and Closures• Exception Handling• Operator Overloading• Introspection, Reflection, Meta Programming

Basics: Variables

key = value

Basics: Variables

agua = 0> 0presupuesto = 0> 0

Basics: Types of Variables

Capitalized variable names are known as constants. Cannot be chaned:CONSTANT = “light speed”


•Constant variables• Local variables•Global variables•Class variables• Instance variables

Basics: Constant Variables

Capitalized variable names are known as constants, and their value should only be assigned once.

Example: Constant Variables


• Constant variables– Cannot be changed.

• Local variables– Local to a specific scope. Such as a method.

• Global variables– Accessible through the entire progarm

• Class variables– Accessible to the class.

• Instance variables– Specific to each instance of a class

Basics: Comment Code

# this is a commentkey = value

Basic Comparison Operators

key == key2 # (equal)key != key2 # (not equal)key > key2 # (bigger than)Key < key2 # (smaller than)key >= key2 #(bigger or equal)key <= key2 # (less or equal)

Basic Comparison Operators

Other Comparison Operators

Example: Comparison Operation

prespuesto == agua> true

Basic: Assignment Operations

Basic: Assignment Operations

Examples: You can store the output

agua + 1> 1agua> 0 agua = agua + 1> 1agua> 1agua += 1> 2

Basics: Logical Operators

Basics: Logical Operators

Basics: Conditionals

if(condition) …end


if(condition and !condition2) …end


if(condition) …else … end


if(condition) …else … end


if(condition) …elsif (condition2 == value) …else … end

Example: Conditionals

Basics: Methods

def method_name …

end

Example: Methods

def say_hiputs “Hi”

endsay_hi> “Hi”

Basics: Methods and arguments

def method_name(argument)…

end

Basics: Methods and arguments

def method_name(argument)…

end

Example: Methods and arguments

Example: Methods and arguments

Basic: Call methods from Methods

Example: Putting it all together

Example: Parenthesis are optional

Classes

Instantiating a Class

Adding instance variable for Class

Adding instance variable for Class

Adding methods to our Class

Adding methods to our Class

Hands-on Experience

• Now we’re going to have a hands-on experience with Ruby

• Let’s see some examples • Let’s modify them real-time based on Q&A • Let’s catch up on some basic Git

Learn Ruby by Example - Follow me to github:https://github.com/mindware/cap_ruby_training.git

Let’s Learn by Example

• Loops– Basics– Hashes– Arrays

• Classes– Instances– Methods– Getters and Setters– Inheritance– Namespaces

Let’s Learn by Example

• Gems• DotEnv and Environment Variables• Rake• Bundler• Internationalization

…Done! What we did:

• We practiced Git for version control• We learned some ruby basics • We saw some cool ruby examples• Now let’s learn about Ruby Gems

Hashes

GEMS

GEMS LABS DEMO

DOTENV

Installating dotenv system-wide

gem install dotenv

Bundling dotenv

Simply add to your Gemfile:

gem ‘dotenv’

bundle install

Create your Secret file

File name: .env Content:DB_PASSWORD=my secretDB_USER=my user

Accesing the value

require ‘dotenv’Dotenv.load

puts ENV[“DB_PASSWORD”] # outputs: ‘my secret’

DOTENV EXAMPLETime for a Demo

Rack is the foundation for all modern Ruby Web Frameworks

Rack provides a common interface between server and Applications.

By wrapping HTTP requests and responses in the simplest way possible, it unifies and distills the API for web servers, web frameworks, and software in between (the so-called middleware) into a single method call.

Rack allows you to write once and run everywhere:• Puma• Goliath• Thin• Webrick

Built for:• Speed• Parallelism• Runs Rack Apps only

Global Installation:gem install puma

Bundler installationgem ‘puma’

Then:bundle install

Run the server: bundle exec puma

Puma powers CAP Web Applications

Command:

puma -t 0:8 -w 4 -p 3000 -e production --preload config.ru

Detailed Demo

• Let’s see some demos and practice:– EventMachine– Goliath– Grape– Sinatra– Padrino– Redis-Rb– Hi-Redis– EM::Synchrony

• Q&A

Detailed Demo

• Let’s see some demos and practice:– EventMachine– Goliath– Grape– Sinatra– Padrino– Redis-Rb– Hi-Redis– EM::Synchrony

• Q&A

Now let’s see how we used these:

• Let’s review our Github Source Code for:– CAP Web App– GMQ CAP API– GMQ Workers

• Head over to:– https://github.com/commonwealth-of-puerto-rico

• Q&A

https://github.com/commonwealth-of-puerto-rico



Topics

• CAP Project Overview• Q&A

Project Overview

• What was wrong• What we did to fix it• What we achieved• Moving forward

PR.gov Infrastructure Topics

• Security Philosophy• Networks Segments• Virtual Router Redundancy Protocol• Documentation

PROJECT OVERVIEW(in Spanish)

Certificado de Antecedentes

Penales y Nuevo App de

PRGOV

En 36 dias:

64,366 solicitudes recibidas

98%

2%

PR.govCompletadas Pendientes

En Menos de 36 dias:

En Menos de 30 dias:

LOS RETOS

• En la prensa se publicaron algunos de los problemas, pero no todos. Tip of the Iceberg.

• El certificado anterior incorporaba información que no habia sido validada con otras agencias.

• Muchos patronos utilizaban información en el certificado que la Policia no habia validado correctamente.

En el antiguo sistema: si se entraba un seguro social inventado, con la información (falsa) de Homero Simpson, la Policia le emitia un certificado sin validar la identidad.

En las profundidades….• Habia personas cometiendo

fraude con estos certificados;• Sistema se apagaba 8 horas para

hacer backup;• Solicitudes no se reintentaban si

ocurrían un fallos básicos en el

• En los intentos de fraude, el Registro Demográfico de la Policía almacenaba información incorrecta y luego no emitía certificado verdadero dueño del seguro social;

• No habia forma de consultar datos de delitos menos graves de la Policia, por lo que los certificados no cumplian con la ley.

Mas abajo: • Los datos se encontraba en 4 bancos de datos

distintos, que no se hablaban entre si. • RCC, el sistema de Justicia que alimentaba el

banco de datos principal de la Policia, seria decomisado en semanas.

• Policia no tenia forma de sincronizar los delitos Graves de forma automatizada y necesitaban ayuda

Y en el fondo…

Retos que enfrentamosAsí sincronizaban los datos

Tribunales y la Policía de Puerto Rico.

Retos que enfrentamos

Toda solicitud de certificado en línea requeria una validación manual. Un promedio de 2 meses de espera para recibir el certificado.

- playbook.cio.gov

- playbook.cio.gov

+ Understand what people need+ Use Data to Drive decisions

Ciudadanía Móvil

77% Mobile First

Los celulares son el principalmedio de acceso al Internet

en Puerto Rico

Ciudadanía Móvil:

- playbook.cio.gov

+ Bring in experienced teams

OPEI

Estrategia

PR.Gov App - Gestor

Policia /Justicia /Tribunales – Registro Criminal

DTOP/NCIC - Identidad

- playbook.cio.gov

+ Choose a modern Techology Stack

Technology Stack

- playbook.cio.gov

+ Default to open

Código Disponible en Github

https://github.com/commonwealth-of-puerto-rico/prgov_cap_webapp/

START

Ready!

Modulos para la Policia

Desarrollo de APIs y Micro Services (RCI) Equipo

Componentes Técnicos

Desarollo de Web App

Desarollo de SistemaDe Mensajeria Gubernamental PR.Gov

Personal Técnico, Agilidad en Contratación y Accesos

ANTES…

…Antes:

…Antes:

…Antes:

…Antes:

…Antes:• En fallas, las solicitudes no se reintentaban automaticamente• No se emitian certificados positivos• No se validaba la información de identidad previo a la emisión• Certificados emitidos en ventanilla no era posible invalidarlos

posteriormente una vez emitido, aún si contenian errores.• Certificado de PR.gov era aceptado por patronos, pero se imprimia en hoja

de papel regular. • Policia emitia en papel especial con un alto costo para la agencia. • La seguridad del papel no era funcional toda vez que si emitian

incorrectamente un certificado, no podian cancelarlo. • Certificado no expiraba.

NUEVO SERVICIO

Validación de Identidad:

• Aceptamos con o sin acentos, mayuzcula o minuscula.

• La información de las agencias es la utilizada. • El sistema tiene inteligencia para detectar posibles

errores en los nombres e identificar apropiadamente.

• Si toda la validación es correcta, se emite certificado. • De requerir evaluación humana, se envia a analista

de la Policia

Certificados en su Email

Retos del Beta• Algunas personas tienen su información

incorrecta en DTOP y estamos colaborando interagencialmente en el particular

• Interesamos incorporar nuevos métodos de validación en PR.gov (licencias de otros países, y pasaporte)

Logros (Alpha)• Consolidamos cuatro sistemas de datos

criminales en uno, adoptando RCI• Validación identidad del ciudadano en DTOP• Integración del registro de ofensores sexuales • Integración de modulo para entrada de los

delitos menos grave de la Policía • Un mismo proceso de validación, para solicitudes

presenciales en las ventanillas de la Policía y en línea en PR.gov

• Funciona en celulares y tabletas• Sistema escalable y con $0 costos de

licenciamiento

Logros (Beta)• Se emiten certificados positivos por primera vez

por PR.Gov• Certificados que antes salían negativos, ahora

salen positivos correctamente. • Se valida la identidad del ciudadano previo a la

emisión.• Se incorporó más allá que la tecnología, un

análisis de los procesos operacionales de la Policía, para atender sus necesidades.

• Servicio en español e inglés

Logros del Prototipo (Beta)• Servicio en español e inglés• Por primera vez, personal que es sentenciado,

busca certificado el mismo día, y se le emite positivo.

Emails: dia antes del lanzamiento

Tráfico: día antes del lanzamiento

Sistema de Mensajería Gubernamental:Transacciones completadas: 893Visitas al GMQ: 14933

Tráfico Móvil

- playbook.cio.gov

+ Address the whole experience, from start to finish.

El Primer App de PR.gov

Disponible para:

Android y Iphone

PRGOV App

Escanea códigos de forma Segura

Solicita tu Certificado

Resumen:

• Nuevo Certificado es más seguro y rápido.• Require un ID de DTOP• Se trámita rápido y de forma segura• Funciona en tu móvil, tabletas y PCs• Versión beta está disponible en:– http://servicios.pr.gov/cap

• App de solicitud y validación disponible para Android y Iphone (keyword: PRGOV)

http://servicios.pr.gov/cap