Upload
william-el-kaim
View
481
Download
5
Embed Size (px)
Citation preview
This Presentation is part of the
Enterprise Architecture Digital Codex
http://www.eacodex.com/Copyright © William El Kaim 2016 2
Plan
• Some Definitions
• Looking for the Next Gen Architecture
• Antifragile Architecture
• Microservice Architecture
• Microservice Technical Ecosystem
• References and Resources
Copyright © William El Kaim 2016 3
A “Service” is software that …
• … is owned, built, and run by an organization
• … is responsible for holding, processing, and/or distributing particular kinds of information within the scope of a system
• … can be built, deployed, and run independently, meeting defined operational objectives
• … communicates with consumers and other services, presenting information using conventions and/or contract assurances
• … protects itself against unwanted access, and its information against loss
• … handles failure conditions such that failures cannot lead to information corruption
Source: Clemens VastersCopyright © William El Kaim 2016 5
• ... “Cloud”
• … “Server”
• … “ESB”
• … “API”
• … XML
• … JSON
• … REST
• … HTTP
• … SOAP
• … WSDL
• … Swagger
• … Docker
• … Mesos
• … Svc Fabric
• … Zookeeper
• … Kubernetes
• … SQL
• … NoSQL
• … MQTT
• … AMQP
• … Scale
• … Reliability
• … “Stateless”
• … “Stateful”
• … OAuth2
• … OpenID
• … X509
• … Java
• … Node
• … C#
• … OOP
• … DDD
• etc. pp.
“Service” does not imply…
Principles
of Service-Based Architecture
are independent of implementation choices.
Source: Clemens VastersCopyright © William El Kaim 2016 6
• A system is a federation of services
and systems, aiming to provide a
composite solution for a well-defined
scope.
• The solution scope may be motivated
by business, technology, policy, law,
culture, or other criteria
• A system may appear and act as a
service towards other parties.
• Systems may share services
• Consumers may interact with multiple
systems
System
Source: Clemens VastersCopyright © William El Kaim 2016 7
Layers: Code Organization
• We usually structure implementation (code)
into several distinct layers.
• Most commonly:
• “Interface” captures information
• Presentation Events
• HTML, GUI, Web Services, Pipes, Queues, RPC, …
• System Events
• Timers, OS Wait Objects, Alerts, …
• “Logic” filters, validates, and processes information
• Functions, Classes, Lambdas, Actors, etc.
• “Resources” are platform
• Web Services, Databases, Queues, …
Interface
Logic
Resources
Source: Clemens VastersCopyright © William El Kaim 2016 8
Tiers: Runtime Organization
• Tiers are about meeting operational objectives
• Aspects of one service or even one layer may have
different scalability and reliability goals
• Resource governance (I/O, CPU, Memory) needs may
differ between particular functions
• UX tier will be more efficient and more adaptable with
client-based rendering
• Tier boundary most often is a process boundary
• On same machine, across machines
• In same organization, across organizations
• In trusted environment, across trust boundaries
• Tier boundaries often cut through layers
• Cuts may separate “yours” and “theirs”
• Ex: “Your” hosted web code and “their” browser
• Ex: “Your” data access code and “their” database
API Gateway
Service Backend
Browser
Web Server
2 tiers, 1 layer
2 tiers, 2 layers
Source: Clemens VastersCopyright © William El Kaim 2016 9
Tier
Tier
Tier
• Tiers have different shapes
• "Fat“ or “Thin“
• Tiers have different relationships
• "Far", “Near" or “Local“; "Connected" or
“Disconnected"
• Focus is on boundaries
• Relation between Layers and Tiers
• Tiers may combine layers or split layers
• Tier != Layer
• Tier: Autonomous runtime entity
• Can be independently operated and
deployed
Shaping Tiers
Layer
Layer
Layer
Layer
Tier
Layer
Layer
Layer
Tier
Layer
Layer
Tier
Layer
Layer
Far, sparsely connected
Near
Local
Far
Source: Clemens VastersCopyright © William El Kaim 2016 10
Tiers: Fat vs. Thin
• “Thin” Tiers
• May implement only parts of layers
• Web Browser provides client portion
of an interactive Web user-interface
layer
• GUI toolkits (Windows Forms, WPF,
Cocoa, etc) may provide client
portion of a GUI/Web services user-
interface layer
• “Fat” Tiers
• Implement full or even multiple
layers
• “Classic” Client/Server typically
implements desktop application with
tier boundary through data layer –
connecting to remote RDBMS
Tier
Tier
Tier
Layer
Layer
Layer
Layer
Tier
Layer
Layer
Layer
Tier
Layer
Layer
Tier
Layer
Layer
Fat
Thin
Source: Clemens VastersCopyright © William El Kaim 2016 11
Tiers: Far vs. Near vs. Local
• Local Tiers• Separated by logical or actual (process)
boundary on local machine.• Logical means: InProc/OutProc via config / no
recompile
• Mandates dynamic loading, factories and activators
• Local Tiers never become “Near” or “Far”
• Near Tiers• Separated by actual process boundary across
machines, inside trust domain and “local network”
• Never become “Far”
• Far Tiers• Separated by trust, ownership and (possibly)
oceans
• May be “connected” or “disconnected”
Tier
Tier
Tier
Layer
Layer
Layer
Layer
Tier
Layer
Layer
Layer
Tier
Layer
Layer
Tier
Layer
Layer
Far, sparsely connected
Near
Local
Far
Source: Clemens VastersCopyright © William El Kaim 2016 12
Layers, Tiers, and Services
• Layers: Code Management
• Tiers: Runtime Management
• Services: Ownership Management
The implementation of a service consistsof one or multiple deployment tiers that
implement one or multiple layers
A “service” is a software and operations deliverable owned by an
autonomous organization.
Source: Clemens VastersCopyright © William El Kaim 2016 13
Three-tier Architecture Lacks Scalability
• Designed in an era where the idea of elasticity and rapid
scaling did not broadly exist.
• Functional components of the application are packaged
together as a unit (the monolith), so the only way you can
respond to changing levels of client demand is to scale the
entire application. Applications running on the three-tier
architecture are typically unable to scale specific pieces of
the application independently because the entire
application is coupled together.
• Regardless of whether you have an e–commerce store, a
social media application, or a blog, a basic requirement for
today’s applications is the ability to scale up and down on
demand; preferably at as low cost as possible.
Source: Andreas SchroederCopyright © William El Kaim 2016 15
Three-tier Architecture Lacks Scalability
Shift of User Interface logic from the server to the client
Source: Octo technologyCopyright © William El Kaim 2016 16
Three-tier Architecture Lacks Scalability
• Still huge codebases (one per layer)
• … with the same impact on development, building, and deployment
• Scaling works better, but still limited
• Staff growth is limited: roughly speaking, one team per layer works well
• Developers become specialists on their layer
• Communication between teams is biased by layer experience (or lack
thereof)
Source: Andreas SchroederCopyright © William El Kaim 2016 17
Four-Tier Engagement Platform
Source: Mobile Needs A Four-Tier Engagement Platform, Forrester report, October 18, 2013Copyright © William El Kaim 2016 18
Four-Tier Engagement Platform
• Client tier
• Packages the unique capabilities of both the experience and the device requesting
information—anything from various mobile clients and wearables, to objects within the
Internet of Things.
• Frees developers from having to customize development to each mobile device, which
allows them instead to focus on building out a single app, increasing productivity, and
decreasing maintenance load.
• Delivery tier
• Optimizes delivery of the digital experience to the user using intelligence received from
the client layer.
• Uses intelligence-driven solutions such as content delivery networks (CDNs) and on-the-
fly optimization tools such as those used for compressing images to decrease bandwidth
• Utilizes sophisticated caching algorithms and tools that enable devops to monitor and
resolve application performance and delivery issues in real time.
Source: Tibco BlogCopyright © William El Kaim 2016 19
Four-Tier Engagement Platform
• Aggregation tier• Serves as the center of application logic, performing tasks like translating between
SOAP to JSON encoding or combining third-party and in-house algorithms to perform complex calculations
• Manages the API layer and facilitates simple service composition.
• This tier connects app requests to the right services with bidirectional, real-time translation to the right data format for back-end and third-party systems, as well as client requests.
• Services tier• Continues to maintain functionality for data both internally and externally.
• By architecting this layer to continuously deploy services, the rate of consumption does not affect the other tiers within the system.
• Without a strong services layer providing the foundation, and ultimately the execution for your application, maintenance and updating can be costly and difficult.
• The services tier is designed for a microservices approach, one that is designed to be open and pluggable, and focuses on the integration and composition of existing services a company has already built as well as new open source libraries.
Source: Tibco BlogCopyright © William El Kaim 2016 20
Four-Tier Engagement Platform – So What?
• The most dramatic difference in this new model is the client tier
• User-facing layer has its own independent set of functionality that leverages the delivery,
aggregation, and services layers beneath it to create device-specific and highly tailored
experiences.
• Isolation gives front-end and user-experience designers much more control to create
memorable digital experiences by tailoring them to the specific user context
Copyright © William El Kaim 2016 21
“Next-Gen” Architecture
• Technology that Scales• Common migrations to {Python, Go, JVM languages}
• Concurrency
• Asynchrony
• Independent systems• Fit-for-purpose systems: analytics, search
• Layered services: caching, etc.
• Event queue
• Scalable persistence• Break up the monolithic database
• Functional partitioning
• Sharding
• Scalable Infrastructure• IaaS for some systems
Looking for the “Minimal Viable
Architecture”
Source: Randy SoupCopyright © William El Kaim 2016 22
Antifragile
• In Antifragile, Nassim Taleb discusses the behavior of
complex systems and distinguishes three kinds:
• those that are fragile: Shatters when exposed to even a
small stressor. Unlike risk, fragility is actually measurable!
• those that are robust or resilient: Fragile with a thicker skin,
vulnerable to unanticipated events
• those that are antifragile: when exposed to stress it gets
stronger .
• These types of systems differ in how they respond to
volatility: “The fragile wants tranquility, the antifragile
grows from disorder, and the robust doesn’t care too
much.”
Copyright © William El Kaim 2016 26
Antifragile While fragile systems are easily injured and suffer from
volatility, antifragile systems grow stronger in response to volatility. So-called robust systems remain unchanged.
Copyright © William El Kaim 2016 27
Antifragility as an Outgrowth of Agile
Source: PWCCopyright © William El Kaim 2016 28
Antifragile: “Black Swan” Problem
• “Black Swan”• Impossibility of calculating the risks of consequential rare events and predicting their
occurrence.
• Taleb proposes that systems should be built to handle Black Swan events – unpredictable
and irregular events of massive consequence – instead of building systems based on
known, predictable patterns.
• Systems are generally purposely designed to deal with known risks so when
a shock to a system occurs that was not predicted, all hell breaks loose.
Copyright © William El Kaim 2016 29
Antifragile: Event Driven Architecture
• Developing software as an aggregation of events that respond to change in
data or state is not a new trend.
• Cloud lets developers the ability to focus on the projects they are doing and
not the infrastructure.
• And Functional reactive programming takes this to the next level.
• By idealizing continuous event streams and building subscribers to the event streams, it
simplifies the idea of event driven systems.
• For developers, it is about minimizing the moving parts of building large scale event
driven systems.
• A definition of a modern online application is now a collection of services that
persist their state outside itself, run independently on independent
infrastructure, can be scaled horizontally and upgraded to avoid minimum or
no downtime to the end user.
Copyright © William El Kaim 2016 30
Anti-Fragile IT Systems
• For many years, the focus in IT has been on building robust systems that
invested heavily in avoiding failures.
• To accomplish this goal, methodical processes were implemented to guide IT
through a list of known use cases so that systems could try to avoid failing
and have a plan for recovery if a failure did occur.
• This approach often led to long delivery cycles and a high degree of
complexity which in reality, increased the risk of failure and created fragile
systems.
• Fragile systems are those systems that cannot adapt to unpredictable,
volatile, and random events often referred to as “shocks to the system”.
• There is a fundamental difference in designing systems that do not fail
versus designing systems that expect all parts of the system to fail.
Source: Mike KavisCopyright © William El Kaim 2016 31
Anti-Fragile IT Systems
• What should be done?
• Assume everything will fail
• Cause failure to validate resiliency
• Test design assumption by stressing them
• Don’t wait for random failure. Remove its uncertainty by forcing it periodically.
• Microservices are a natural design approach for Antifragile
• Gives you the full power of embracing change as you are able to evolve parts of your
application that change at similar rates, typically microservices, at the rate at which they
need to evolve at.
• Enable clients must respond gracefully to provider failure
• Devops and Aggressive monitoring
• Try to break your IT systems using techniques such as game days and systems
like chaos monkey.
Copyright © William El Kaim 2016 32
Netflix Chaos Monkey
• “One of the first systems our
engineers built in AWS is called
the Chaos Monkey.
• The Chaos Monkey’s job is to
randomly kill instances and
services within our architecture.
• If we aren’t constantly testing our
ability to succeed despite failure,
then it isn’t likely to work when it
matters most – in the event of an
unexpected outage.”http://luckyrobot.com/netflix-chaos-monkey-keeps-movies-streaming/http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html
Copyright © William El Kaim 2016 33
Then, Netflix Simian Army
• Simian Army consists of services (Monkeys)
in the cloud for generating various kinds of
failures, detecting abnormal conditions, and
testing our ability to survive them.
• The goal is to keep our cloud safe, secure,
and highly available. More details can be
found at this blog.
• Currently the simians include Chaos
Monkey, Janitor Monkey, and Conformity
Monkey.
• Refer to the Quick start guide to get started
setting up and using the Monkeys.
https://github.com/Netflix/SimianArmyCopyright © William El Kaim 2016 34
Conway’s Law
• “Organizations which design systems ... are constrained to produce designs
which are copies of the communication structures of these organizations”
Melvin Conway
• Siloed functional teams lead to siloed application architectures
• Objective: Do the opposite
• Create cross-functional team organized around capabilities and responsibility
• Fight against the monolith …
Source: Neal Ford, ThoughtWorksCopyright © William El Kaim 2016 36
Software Monolith
• A Software Monolith
• One build and deployment unit
• One code base
• One technology stack (Linux, JVM, Tomcat,
Libraries)
• Benefits
• Simple mental model for developers
• one unit of access for coding, building, and
deploying
• Simple scaling model for operations
• just run multiple copies behind a load balancer
Source: Andreas SchroederCopyright © William El Kaim 2016 37
Software Monolith Issues
• Huge and intimidating code base for developers
• Development tools get overburdened
• refactorings take minutes
• builds take hours
• testing in continuous integration takes days
• Scaling is limited
• Running a copy of the whole system is resource-intense
• It doesn’t scale with the data volume out-of-the-box
• Deployment frequency is limited
• Re-deploying means halting the whole system
• Re-deployments will fail and increase the perceived risk of deployment
Source: Andreas SchroederCopyright © William El Kaim 2016 38
What is a Microservice Architecture?
• Definitions
• Functional system decomposition into manageable and independently deployable
components.
• The term “micro” refers to the sizing: a microservice must be manageable by a single
development team (5-9 developers)
• Functional system decomposition means vertical slicing (in contrast to horizontal slicing
through layers)
• Independent deployability implies no shared state and inter-process communication (often via
HTTP REST-ish interfaces)
• Enables separation and independent evolution of code base, technology stacks, scaling and,
features.
• Loosely coupled service oriented architecture with bounded context.
• Microservice is the first architectural style developed post-Continuous
Delivery
• Each service has its own software repository
Copyright © William El Kaim 2016 39
Microservice History
• 2011: First discussions using this term at a
software architecture workshop near Venice
• May 2012: microservices settled as the most
appropriate term
• March 2012: “Java, the Unix Way” at 33rd
degree by James Lewis
• September 2012: “μService Architecture“ at
Baruco by Fred George
• All along, Adrian Cockroft pioneered this style
at Netflix as “fine grained SOA”
Source: Andreas SchroederCopyright © William El Kaim 2016 40
Principles of Microservice
Source: PWC
Decentralized Governance: Enterprise
Architects suffer from less pressure to
make the correct choice(s) in microservice
architectures.
Source: Andreas Schroeder
Copyright © William El Kaim 2016 41
Monolithic vs. Microservices Architecture
Source: PWCCopyright © William El Kaim 2016 43
Monolithic vs. MicroServices Architecture
Source: Neal Ford, ThoughtWorks
Small enough to fit in your head
Rewrite over maintain
(10—1000 LOC)-ish / service
Copyright © William El Kaim 2016 45
Monolithic vs. SOA
Source: PWC
Source: Neal Ford, ThoughtWorks
Copyright © William El Kaim 2016 46
MSA = SnowMan Architecture
From Horizontal to Vertical decomposition
Source: The Snowman architectureCopyright © William El Kaim 2016 47
Microservice Architecture Pattern
Source: Microservices.ioCopyright © William El Kaim 2016 48
Examples
• eBay
• 5th generation today
• Monolithic Perl Monolithic C++ Java microservices
• 3rd generation today
• Monolithic Rails JS / Rails / Scala microservices
• Amazon
• Nth generation today
• Monolithic Perl / C++ Java / Scala microservices
Re-architecting is a sign of success!
If you never need to, either you
overbuilt or nobody cares.
Source: Randy SoupCopyright © William El Kaim 2016 49
MSA Drawbacks
• Complexity is moved not removed
• Understanding, managing and testing dependencies is difficult
• Centralized governance is not possible
• Large numbers of Microservices are difficult to orchestrate
• Increase network communication
• Independently running component interact with each other using calls. Such system
require reliable and fast network connections.
• Network Security
• Inter-service communication need to be secured to avoid any security breach
• Microservice based application are more prone to security vulnerabilities.
• Production Monitoring
• Monitoring the application in production becomes a complex job, with multiple services
• Usage of Containers could help
Source: WSO2Copyright © William El Kaim 2016 50
MSA Ecosystem
• Compared with traditional packaging methods for monolithic applications,
microservice architecture gives you significantly more flexibility in terms of
deployment options and scalability configurations.
• However, microservice architecture also underscores the need for a robust
automation infrastructure; given the vast number of moving parts, the
prospect of deploying and managing microservices manually is simply
untenable.
• Microservice architecture increases the complexity of coordination, error
handling, monitoring and state management across services.
• The cost of achieving extreme scalability and resiliency in this architecture
places a burden on architects to understand the implications of decoupling,
of increased independence between services, and of apps and applications
that must aggregate and consume a larger number of services.
Copyright © William El Kaim 2016 52
MSA: Agile Development & Devops
Scale enable elastic Cloud Architecture and devops techniquesSpeed enables and encourages new microservice architectures
Scale breaks HardwareSpeed breaks Software
Source: Adrian CockcroftCopyright © William El Kaim 2016 54
MSA: Deployed in Containers
Source: PWCCopyright © William El Kaim 2016 55
Microservice: Container
• Dedicated Solution
• Cap Gemini Apollo / Mantl / Mesos / NOMAD / Terraform
• Container
• Docker / Amazon EC2 Container Service / Apache Karaf
• Other container tools
• Deis: open source PaaS that makes it easy to deploy and manage applications on your
own servers.
• Packer: tool for creating identical machine images for multiple platforms from a single
source configuration.
• Google Kubernetes: open source orchestration system for Docker containers
• Container specific OS
• Snappy Ubuntu Core (Snappy)
• RedHat Atomic
Copyright © William El Kaim 2016 56
Microservice Communication Protocol: gRPC
• Thanks to the rise of JavaScript frameworks, Node.js, and document
databases, REST has become wildly popular among web developers.
• Many applications started to rely on REST even for internal serialization and
communication patterns.
• But is HTTP the most efficient protocol for exchanging messages across
services running in the same context, same network, and possibly the same
machine?
• HTTP’s convenience comes with a huge performance trade-off, which takes us back to
the issue of finding the most optimal communication framework for microservices.
• Enter gRPC, the modern, lightweight communication protocol from Google.
• It’s a high-performance, open-source universal remote procedure call (RPC) framework
that works across a dozen languages running in any OS.
Source: GRPCCopyright © William El Kaim 2016 57
Microservice Communication Protocol: gRPC
• Declare the service in a language-agnostic Interface Definition Language (IDL), and then generate language-specific bindings
• Make the clients believe that the server is on the same machine. • Clients invoke a method on the Stub, which gets transparently
handled by the underlying protocol.
• gRPC’s secret sauce lies in the way the serialization is handled. • Based on Protocol Buffers, an open source mechanism for
serializing structured data, which is language and platform neutral.
• The latest version of Protocol Buffer is proto3, which supports code generation in Java, C++, Python, Java Lite, Ruby, JavaScript, Objective-C, and C#. When a Protocol Buffer is compiled for a specific language, it comes with accessors (setters and getters) for each field definition.
Source: GRPCCopyright © William El Kaim 2016 58
Microservice: Frameworks
• DropWizard (Java)
• Gizmo
• Kite (Go)
• Micro
• NancyFX (.net and Mono)
• Playframework (Java & Scala)
• Qbit (Java)
• Rodent (Ruby)
• Seneca: micro-services toolkit for Node.js
• ServiceStack (F#)
• Spring Boot (Java)
• Vert.x (Java, JavaScript, CoffeeScript, Ruby, Python or Groovy)
Copyright © William El Kaim 2016 59
Microservice: Platform and Automation
Source: Adrian CockcroftCopyright © William El Kaim 2016 60
Microservice: Platform and Automation
• Platform for microservices• Colossus / Gilliam / Hivepod / Lagom / Netflix OSS / Silex
• Deployment automation• Jenkins, uDeploy, Capistrano, Chief, Puppet or custom scripts.
• Distributed tracing and context propagation• Circonus, Nexflix Suro, OpenTracing, Prometheus, Riemann.io, Sensu, Trace, Zipkin
• Application Performance Monitoring• AppDynamics, Datadog, New Relic, Scout, SignalFx, and Sysdig
• Latency and Fault Tolerance• Hystrix
• Security• Open Source Identity & Access Management OSIAM
• VAddy (http://vaddy.net): Continuous web security testing service connected with CI tools.
Copyright © William El Kaim 2016 61
Lagom
https://www.lightbend.com/lagomCopyright © William El Kaim 2016 62
MSA Platform Ex: Netflix OSShttp://netflix.github.io/
Source: Adrian CockcroftCopyright © William El Kaim 2016 64
MSA Platform Examples
Twitter Microservice Platform
Gilt Microservice Platform
Source: Adrian CockcroftCopyright © William El Kaim 2016 65
References: Microservice Introduction
• Introduction to Microservice by Martin Fowler.
• Introduction to Microservices by Nginx.
• InfoQ series of articles
• David Morgantini series of Blog Posts
• Microservice Architectures, Dr. Andreas Schroeder
• High Scalability Article
• Microservices: The resurgence of SOA principles and an alternative to the
monolith, PWC
• Why a microservice approach to building Application by Microsoft.
• State of the Art in Microservices, Adrian Cockcroft and Microservices: State
of the Union by Adrian Cockcroft.
Copyright © William El Kaim 2016 67
Best Practices & Lessons Learned
• Microservice.com: Pragmatic advice for microservices practitioners.
• Microservice Architecture Pattern.
• Minimimum Viable Architecture in startup, Randy Soup
• It’s Time to Move to a Four-Tier Application Architecture, Nginx
• Developing applications with a microservice architecture, Chris Richardson
• Sam Newman’s 14 Tips For Microservices, ThoughtWorks
• Building SaaS enabled architecture, Twelve Factor App
• Building PaaS enabled architecture, ActiveState Blog
• Failing at Microservice by Richard Clayton
• Microservices Reality Check, CapGemini
• Migrating to Microservices and Slides, Adrian Cockcroft
• Seven microservices architecture problems and solutions, Eugene Dvorkin
• bla bla microservices bla bla
Copyright © William El Kaim 2016 68
Microservice: Case Studies
• Microservice Architecture by Netflix
• Using Services to Break Down Monoliths by Yelp
• Adopting Microservices at Netflix
• Effectively Monitor Your Micro-Service Architectures by wheretoe.at
• A Journey into Microservices by Hailo
• Scaling Gilt: from Monolithic Ruby Application to Distributed Scala Micro-
Services Architecture by Gilt
• I-Tier: Breaking Up the Monolith by Groupon
• What has Soundcloud learned about Microservices? by Soundcloud
• Microservices from Theory to Practice: Creating Applications in IBM Bluemix
by IBM
Copyright © William El Kaim 2016 69
Microservice: Books and News
• News
• Microservice Weekly
• The New Stack
• StackShare
• Books
• “Antifragile Software: Building Adaptable Software with Microservices” by Russ Miles
• “Building Microservices: Designing Fine-Grained Systems” by Sam Newman
• “Drift into Failure: From Hunting Broken Components to Understanding Complex
Systems” by Sidney Dekker
• “The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win” by
Gene Kim, Kevin Behr, & George Spafford.
Copyright © William El Kaim 2016 70
http://www.twitter.com/welkaim
SlideShare
http://www.slideshare.net/welkaim
EA Digital Codex
http://www.eacodex.com/
http://fr.linkedin.com/in/williamelkaim
Claudine O'SullivanCopyright © William El Kaim 2016 71