Enterprise DevOps - Citihub

Integrity.Excellence.Results.

Enterprise DevOps:How to Maintain Segregation of Duties with CICD

www.citihub.com

Contents

3 Introduction

4 Understanding the Context

5 CICD Workow and Core Roles

7 Roles Matrix

8 Control Points

9 How to Maintain Automation

10 10 Conclusion

11 About Citihub Consulting

AuthorErhan Sen, Senior Consultant

Erhan is an innovative senior technology leader with many years of experience in design, development and

end-to-end delivery of large-scale banking and financial systems. He joined Citihub from Capgemini and has been

responsible for formulating, leading and governing enterprise architecture roadmaps, blueprint activities and

associated strategic technology transformations. Erhan has deep expertise in DevOps and CICD automation.

10

Introduction

DefinitionsSegregation of Duties (SoD)

Infrastructure as Code (IaC)

Segregation of duties (also known as

"separation of duties") is the concept of having

more than one person required to complete a

task. In business, this is an internal control

intended to prevent fraud and error. For

Financial Services firms, some regulators

specifically issue firm guidance around the

needneed for SoD in the software delivery lifecycle.

Here are a few examples of references made to

SoD:

o COBIT 5 -> APO01.02

o OSA -> AC-5, AC-6

o ISO 17799 -> 10.1.3, 10.6.1, 10.10.1

o PCI-DSS v2 -> 7.1.2

o NIST 800-53 -> AC-5

o SOX 404, referenced by list of accepted

internal controls

o MAS-TRM -> 7.2.4, 11.0.1.b, F.2.2.a

InfrastructuInfrastructure as Code (IaC) is the process of

managing and provisioning computing

infrastructure (processes, bare-metal servers,

virtual servers, etc.) and their configuration

through machine process-able definition files,

rather than physical hardware configuration or

the use of interactive configuration tools.

•

•

The cost and time-to-market benefits of DevOps and CICD are well advertised but the benefits in helping to manage risk are less

well understood. As discussion develops towards CICD pipelines that are heavily automated and preparing critical code for

deployment in short cycles, the first reaction from risk managers and delivery executives can be one of fear.

In this paper, we explore one of the main fears: that segregation of duties will be compromised.

The practice of integrating, building, testing

and releasing code in short cycles with the help

of automation

ContinuousContinuous Integration: integrating code into a

shared repository frequently (preferably at least

once a day). Each check-in is then verified

through an automated build, allowing teams to

detect problems early

ContinuousContinuous Delivery: Preparing the release

build for production by automated validation

and verification techniques

Note:Note: Although it is out of scope of this article,

the reader should understand that Continuous

Deployment goes one more step further than

Continuous Delivery so that an automatically

verified and prepared release build is also

automatically deployed to production

environments along with any related

infrastructuinfrastructure changes

•

•

•

•

CICD: Continuous Integration (CI) / Continuous Delivery (CD)

Enterprise DevOps: How to Maintain Segregation of Duties with CICD

3

Understanding the Context

Given that the intention of CICD is to automate and speed up

the process of delivering code, there is a genuine risk that a

single individual could be given the ability to develop, compile

and move code from one environment to another. And hence

either maliciously, or by error, introduce risk to a firm’s

production environment. Moreover, a loophole or mistake can

expose confidential information, or cause significant

downtime downtime resulting in reputational or financial loss for the firm.

Although DevOps marketing promotes freedom and

efficiency, it is not the intention of CICD to give a single

developer all this power. The risks can be avoided with the

proper controls in place without compromising too much on

efficiency and automation.

•

•

•

•

Understanding the Risks

Having said that, regulators and auditors want to see proper

implementations which mitigate risks rooting from conflicts of

interest and excessive rights. Apart from employee

entitlements and separation of toxic combinations of roles,

there are other requirements that need to be observed in

DevOps systems in order to complete the whole SoD picture

including:

Proof of proper controls and justification of any type of

change in the system against an approved change

request are fundamental controls from an auditor’s

perspective.

Each action in the process should be designed carefully

so that they can receive, pass and log change

information.

AuditorsAuditors require audit trails throughout the delivery

process, bringing traceability and auditability

requirements into the picture.

Finally, auditors will question the implementation,

currency and integrity of governance rules and policies to

measure adherence to compliance, standards and best

practices.

Understanding the Regulators and Auditors


4

Roles

Quality Gate/Check Point/Approval

Operations Admin

DeveloperTester

Peer or Lead Developer

Build Engineer

Infrastructure/Database Admin

DevOps Tool Admin

Coding Guidelines Architecture Standards Risk & Compliance Policies Training Materials

CodeIntegration

SecurityScan

Build

BinaryRepositoryMng

Open SourceManagement

Code Quality& ComplianceScan

Change Management

RequirementsGathering Design Configuration & Environment Management

Deployment

Fault Detection

Incident Management

ProjectPlanning

AnalysisReview

Coding

Task/IssueAssignment

Test CaseDevelopment

Code Review(Peer)

PerformanceTesting

UnitTesting

Verification & Validation

Knowledge Management, Enterprise Governance Continuum & Collaboration

RegressionTesting

UX Testing FunctionalTesting

UI Testing

Release Management & Pipeline Automation

QG1 QG2 QG3 QG4 QG5 QG6

ProjectInitiation & Planning

Analysis DevelopmentIntegration Q&A UAT

Validation & VerificationProduction

Audit RepositoryDevOps Intelligence

Management Dashboards Performance Metric Repository

Plan Deliver Service

Continuous Deployment

Continuous Delivery

Continuous Integration


Citihub’s DevOps reference model is designed to bring compliance to regulators’ SoD requirements without sacrificing agility and

speed. The picture below depicts a workflow with quality gates and a distribution of roles which can achieve our goals.

5

CICD Workflow and Core Roles

Developer: Creates or modifies code and is responsible for

requesting the release of their components into production.

Developers should not be able to manipulate a system,

database, test results, DevOps tool configuration or DevOps

process outside of the development environment. Any

environment higher in the stack, such as testing and

production, should be considered off limits for the

development teams’ didevelopment teams’ direct access.

Tester: Develops test cases as well as automated functional

tests. Testers should not have rights to develop or deploy

code, manipulate collected test results, manipulate a system,

database, DevOps tool configuration or DevOps process.

Code Reviewer: As a second pair of eyes to review code to

identify non-compliant or malicious code. Developers should

not be able to review or approve their own code.

BuildBuild Engineer: For each deployable package this role

designs, develops and configures the process which checks

the code out from the version control system and outputs a

deployable package. It is uncommon to have a dedicated

individual performing a build engineer role, but combining the

role with other roles especially with developer role, introduces

risk. The role is critical to ensure standardization and

adheadherence to security, risk and compliance policies. However,

the role has the power to manipulate results, rules and tasks

associated with continuous integration process flow. The

build engineer role should not be given to developers.

Moreover, build engineers should not have write access to

source code repositories. If developer and build engineer

roles are combined, additional compensating controls are

rrequired for example, a second reviewer should be

responsible for review and approval for the build process

configuration.

Infrastructure Engineer: Develops virtual infrastructure

components (infrastructure as code - IaC) that can later be

deployed into production. Combining this role with roles that

have write access to production carries risk in the same way

that allowing developers, with write access to source code,

the ability to deploy code into production. Where possible,

developers should not be given this role and infrastructure

engineersengineers should not be given the ability to deploy into

production. The combination of code development and

system configuration is extremely powerful; therefore, these

roles must be separated. If an organization is small or

separation of these roles will bring high cost, there should be

other controls to be in place for eliminating the mentioned

risks.

DevOps Tools Administrator: Responsible for the

configuration of tools and user role mapping in the DevOps

pipeline. Release process configuration, tool configurations

and governance of rules implemented on tools are managed

by this role. This role should be isolated from any type of

development and infrastructure related activities, as the

combination of being able to produce new code and manage

thethe tools that promote code into production would afford a

single user the ability to compromise your application or

infrastructure.

Operations Engineer: Responsible for configuration

management, provisioning and deployment of assets to

environments. In traditional architectures, they design and

develop scripts to manage operational tasks. In a DevOps set

up which includes an automated integrated deployment

application, this role designs deployment processes and

configures related deployment environments on the tool. This

rolerole should be isolated from any code development related

and DevOps administration activities.

Role Distribution and ResponsibilitiesTo ensure segregation of duties within the release process, roles should be distributed to different parties dividing the accountability

for creation of code and the distribution and installation of that code. The priorities should be to ensure least privilege and prevent,

or at least, detect attempts to promote unsafe or malicious code. Yet, not all roles are critical, more emphasis is placed on roles

which can directly manipulate code, environments and controls. These roles are listed below.


6

Roles Matrix

The following figure represents a roles matrix, sometimes called a toxic combination matrix, to show the segregation of duties risks.

Roles

Coding

Code Review

Design Functional Test Case

Design Build Process

Run Build Process

Run Any Test

Attest QA Results

DevOps Tools Configuration

Manual Deployment

Automated Deployment Request

Change/Write Production Systems

Tasks

Developer

Code Reviewer

Tester

Build Engineer

DevOps Tool Administrator

Operations Engineer

Infrastructure Engineer


7

Control Points

QG1 – Analysis and Test Case Review: At this stage, the analysis document is evaluated with respect to coverage,

completeness, business rules and test cases. The Business

Analyst Lead, QA Lead or peers can review and approve them

for further implementation.

QG2QG2 – Code Readiness: : Automated Code Review and Peer Review are the two main pillars of this quality gate. Meticulous

machine and/or human inspection of the code (software or

IaC configuration item) to check adherence to coding,

architecture, security and performance guidelines.

Additionally, by collecting unit test results from developer’s

development workbench at this quality gate, this information

cancan be used as evidence to verify a developer code’s

compliance to policies requiring them to testing prior

deployment.

QG3 – Continuous Integration: This quality gate includes performing several series of detailed tests and

measurements. Unit, regression, security, performance,

functional, load and open source compliance tests can be run

and greatly help to achieve DevOps goals. There are two

potential types of actions that can be taken in the event of a

violation or failure, depending on an organization’s policies.

Either the process can be suspended, pending a fix or it can

be continued with a proper waiver by debiting this issue to a

technical debt account. With an extensive control set and a

variety of tests at this quality gate, some organizations may

choose to solely rely on the results here for the decision to

promote code into production.

QG4QG4 - IT QA: This quality gate is present in most enterprises, manual functional, UI and UX tests are performed. This stage

is IT’s last test activity in the whole process.

QG5QG5 - Final Release Assessment: This is the final quality gate before production. It is possible to feed previous

measurements, results, task acknowledgements, approvals,

and waivers via an automated process to the release

manager, team leads and IT managers so that they can judge

the risk of a release.

QG6QG6 – Post Production: After deployment to production environments, production problems are monitored and

collected. Depending on the number and severity of issues,

there is a decision point here to potentially rollback to the last

known healthy state.

Citihub recommends 6 quality gates through the flow of the DevOps delivery pipeline. All 6 quality gates can be automated to some

extent or be fully manual depending on an organization’s maturity, culture, risk appetite and application portfolio.

Quality Gates


8

How to Maintain Automation

The mix of manual controls vs. automated controls will be very

dependent on each organization’s maturity, attitude to risk,

regulatory environment and the Software Development

Lifecycle style in use. An optimal balance between ensuring

that risks are fully mitigated and maximizing the benefits of

DevOps principles (faster time to market etc.) will require

buy-in and approval from a number of stakeholders in the

ororganization. Some may argue that manual controls can be

more robust and that a human can catch things that

automation simply cannot; however, automation improves

standardization, controls reliability and can be complimentary

to manual testing.

One of the biggest things that DevOps brings to delivery and

release management is high standardization with automation

yielding strict enforcement of policies—this is highly desired

by management and auditors. Our model, while automating

the process as much as possible, aims not to undermine the

manageability and audit requirements. Especially, at “Quality

Gate 3 – CI”, there is significant automation and verification

datadata collection. CI tool controls the automated chain. Human

intervention can be eliminated, or at least minimized, by CI’s

role and SoD compliance will improve. When incorporating

multiple testing and scanning tools, highly reliable,

dependable and detailed test results can be achieved. It is

even possible to achieve nearly 100% test coverage after

investing in culture changes, tools and unit tests. This set of

contcontrols improves transparency and identification of issues

without relying on human insight in most scenarios. In case an

unacceptable condition is spotted during tests, the delivery

process can be suspended for a remediation by a human

being.

Another area that automation can be deployed to is the

process execution platform by utilizing an application release

automation (ARA) tool. ARA tooling can be deployed to give

extensive data collection throughout the DevOps pipeline.

Configurations cannot be adjusted in production and will

be mirrored throughout the staging environments

simplifying problem management and removing the risk of

erroneous or malicious changes

TheThe latest infrastructure and security code will be built into

each release. Considering the concept of small, frequent

releases, the implication is that patch management

becomes a continuous process rather than something

requiring significant effort, and the risk of relying on

end-of-life components is greatly reduced.

•

•

An ARA tool or a process execution platform would drive the

entire release process to all environments with solid rules and

flows which guarantee compliance. It makes decisions based

on QG results and SDLC metrics to move forward or raise an

issue and request human intervention. It carries change,

release, task and build IDs throughout the pipeline to create

end-to-end traceability, which as we mentioned above is one

ofof the most important audit requirements. When automation is

not possible or a waiver is needed, the tool collects

information electronically and populate the release repository,

thus augmenting valuable audit trails for risk and compliance.

IT managers and internal auditors can enjoy another huge

benefit as an outcome of intensive data collection which gives

a clear visibility of delivery pipeline internals, and provides

invaluable metrics for further possible optimization. invaluable metrics for further possible optimization.

Finally, when CICD and rapid releases are considered in

conjunction with container technology, enterprises have the

opportunity to take a step change improvement in risk

management. If the enterprise adopts the concept of

immutable infrastructure, which is an approach to managing

software deployments wherein components are replaced

rather than change, i.e. an application or services is effectively

re-deployedre-deployed each time a change occurs, the following

additional benefits can be reaped:


9

Conclusion

We understand concerns about DevOps and its effects on controlling the environment but

automation can also help to reduce risk. Every organization needs to spend time to fully

understanding the risks of a DevOps approach in their environment weighing the results they

want to achieve in order to fit with its culture, risk appetite and circumstances.

Automation is not evil.

Automation not only provides agility and efficiency, but is also capable of decreasing risk and

increasing security when those requirements are designed into the workflow.

Combine these benefits with the traceability advantages of well-implemented CICD, and it is a

future state worth striving for.

Automated DevOps pipelines enforce standardization

Embedded compliance rules can guarantee adherence to internal and external policies

Integrated code and security scans can fortify IT landscape by identification of

malicious code

• • •


10

Contact Us

About Citihub Consulting

EMEA

Richard Hamstead

[email protected]

Moor Place

1 Fore Street

London EC2Y 9DT

+44 800 028 1901+44 800 028 1901

Bellerivestrasse 201

8008 Zurich

+41 44 562 7101


Citihub Consulting is a global, independent IT advisory firm with deep domain expertise across every layer of the

technology stack – from business applications and data platforms down to core infrastructure. From IT strategy,

architecture and solution development, through to cost optimisation, risk assessment and implementation – our

trusted experts deliver the right results for your business.

ForFor us consultancy is personal. We have a relentless commitment to great execution, integrity and client success.

We aim to redefine perceptions of our industry and our commitment to delivering the right results for our clients

has never changed, even as the business has grown consistently over the last decades.

For more information, please visit www.citihub.com

North America

Keith Maitland

[email protected]

500 Fifth Ave, Suite 1610

New York, NY 10110

+1 646-780-1150

The Dineen BuildingThe Dineen Building

140 Yonge Street, Suite 200

Toronto, Ontario, M5C 1X6

+1 437 886 8390

Asia Pacific

Steve Rutherford

[email protected]

3 Pickering Street

#01-64

Singapore 048660

+65 3152 2777+65 3152 2777

Flat 406-9, 4/F

Three Pacific Place

1 Queen’s Road East

Hong Kong

+852 8108 2777

11

Documents

Enterprise DevOps - Citihub