Chasing code quality in huge multi-location team project

April 9, 2023 www.ExigenServices.com

CHASING CODE QUALITY IN HUGE MULTI-LOCATION TEAM PROJECT

2 www.ExigenServices.com

GLOBAL EXIGEN SERVICES

EuropeEastern Europe

US

RigaR&D Center

VilniusR&D Center

Asia-PacificWuxiR&D Center

SuzhouR&D Center

London

Frankfurt

New YorkSan FranciscoHeadquarters

San Paulo

Christchurch

Johannesburg

KazanR&D Center

Nizhniy NovgorodR&D Center

St. PetersburgR&D Center

MinskR&D Center

OdessaR&D Center Dnepropetrovsk

R&D Center• Founded in 2000• 1’500 professionals• Revenue approx.

$70M

Adelaide


EXIGEN SUITE COMPONENTS

3rd Party Interface

BillingInterface

ClaimsInterface

Commission Interface

Self-Service Portal Interface

Reinsurance Interface

Analytics Interface

3rd Party Mgmt

Accounts & CRM

CorrespondenceManagementSub-LedgerReporting &

Compliance Tasks &

Decisions

Document Generation

Configuration & Scheduling Security Process &

Task MgmtBusiness Activity

MonitoringContent

ManagementBusiness

Rules

CommonServices

Data & Sys Integration

Operations Management

New Business

Policy Servicing

Claims Financials

Claim Processing

BusinessProcesses&Core Functions

New Business

Rating

Policy Servicing

Adjudicaton & Settlement

Claim Financials

FNOL

Claim Processing

ProductManagement

Under-Writing

POLICYCORE CLAIMCORE

Payment Processes

Payment Management

Invoice & Collection

Bill Plan Management

BILLINGCORE

PRODUCT FACTORY

Distribution Management Sales and Customer Service DISTRIBUTIONCORE


WHEN YOU START FAILING

Agile works says: One location, team, scrum master and product owner Everybody are universal and senior;

But in sometimes we have Multi location, teams, scrum masters and PO’s And mixed level;

And here problems arise.


THIS PRESENTATION TOPIC

Stable Continuous Integration

Effective Code review

Useful Static code quality


TECHNOLOGIES WE SELECTED

Source control (DSVC) - hg (Mercurial) mercurial.selenic.com

Build tool - Jenkins jenkins-ci.org

Review tool - Reviewboard reviewboard.org

Static code quality - Sonar sonarsource.org


STABLE CONTINUOUS INTEGRATION


CI FOR ONE TEAM


CI WITH MULTIPLE TEAMS


WHAT TO DO?

Options

Fight with continuous integration failures every day

Fail fast. Setup your continuous integration to no accept failing code.


FAILING FAST

“Failing fast” for development means getting response about your code quickly, from: compilerdeploymentunit and integration testUI testsstatic quality parameters

And all those answers should be given before code gets to source control.


MULTIPLE TEAMS – MULTIPLE CONTINUOUS ENVIRONMENTS


TEAM BRANCH


BENEFITS

Team continuous integration environment is effected only by team changes.

Only finished and tested features will get to central. Gatekeeper would be responsible to merge.


STILL TO BE IMPROVED

There is still possibility that gatekeeper will push bad code

We don’t reach our ultimate goal - STABILITY of central.


ONE STEP FURTHER - TEAM-MERGE

To achieve our goal we created CI job that doesn't allow bad code. It does automatic code merge, compile, test and push.– rights write to central are revoked for anyone and only "special"

"team-merge" user can do merge.– only possible way to commit code from team-branch to central

is through "team-merge" script.


HOW IT WORKS


HOW TO START IT


FINAL TIPS

Your environment should be stable enough to do this approach.

Better have small number of working test rather than huge number of failing test.

Each team has own CI plans – hardware is required


EFFECTIVE CODE REVIEW


CODE REVIEW OR NOT CODE REVIEW

Code review, if: – You plan long life cycle for your product– Your customer insist on high quality and you have maintenance

contract.

Don’t code review, if:– Your product has short life cycle (demos, POC, etc.)– You don’t care about maintenance after it’s done


CODE REVIEW

There are various proposals on code review – Pair Programming

Review about do less than 100%, with rules, likeReview only Selected commits.Review only commits from juniorsRandom reviews

Review 100% of code

We chose 100% review.


CODE REVIEW AFTER COMMIT

Guarantee that reviewed code is committed

No special actions required if reviewed code passes

Issues: In case code doesn’t pass you have bad code in repository until it’s

fixed. Bad code effects others


CODE REVIEW BEFORE COMMIT

Review patches or review branch.

Bad code doesn’t effect others.

Issues: You have no guarantee that good code reviewed will get to

repository. As somebody needs to merge it.

What we needed is:

Automated merge on code review pass.


100% CODE REVIEW

What we want is

1. Work as a team.

2. Review all code changes

3. Do not effect others

Therefore we need code review systems what would assure 100% review.

Other requirements: DVCS-backed review system Acts as a front end for to central DVCS-repo When review is complete auto merge with default branch


CODE REVIEW TOOLS

For git users there is good tool called gerrit http://code.google.com/p/gerrit/

If you use mercurial (hg), it’s bit more complicated – as where is no analogue for gerrit for hg.

Our approach was to use reviewboard.org and mercurial enhanced with python scripts what are mimicking gerrit approach.

To initiate review you should call “hg postreview”

http://code.google.com/p/gerrit/




REVIEWBOARD


REVIEW CYCLE


USEFUL STATIC CODE CONTROL


CONTROL CODE QUALITY

We want to control basic parameters of code qualityStatic code violationCode unit and integration test coverageOther code parameters

We can do it effectively with tool called Sonar


METRICS

What you want to measure? – Test Coverage?– Static code violations?– Code metrics?

Answer to this is SONAR

Automates metrics and creates reports.

Allows to analyze code to the code line details.

sonarsource.org


SONAR PROJECT VIEW


CODE COVERAGE


CODE DRILLDOWN


CODE VIOLATIONS


VIOLATION DRILLDOWN


WHAT’S NEXT WITH SONAR

Good is that we have tool to control

Bad is that what people are still people and tend to do crappy code.

number of violations is increasing.still repeating same errors.code is unmaintainable


OUR APPROACH

Decide that violations are critical for you and what you can live with.

What we do: Using multistage commit we don’t accept code with blocker

violations. Commit is declined and developer should rework their code

before next commit. Use personal violations view to work with particular developers

coding style.


PERSONAL VIOLATIONS DRILLDOWN


PREVENT VIOLATIONS


TIPS

Declare violations as critical first.

Allow teams to fix them.

Increase severity to blocker after all existing are fixed

Decline new violations of same type


QUESTIONS?

Documents

Chasing code quality in huge multi-location team project