FME Development

&

Continuous integration

GIM

Smolders Steven

Van Oudenhove Jean-Pierre

Overview

• FME Project types

• FME local processes

• FME centered services

• Multi technology integrated applications

• What is Continuous Integration?

• Continuous Integration applied to FME

• Conclusion

FME local processes

Output Data Input data

FME local processes

• One single developer

• FME desktop

• The focus is the output data

• Run within an isolated or stable environment

• Single Technology - No integration problems

• Maintenance by single developer

FME centered services

Output Data Input data

FME centered services

• FME workspace authored on desktop environment

published to server

• Exposed as services (Web Services, sheduled jobs)

• Components are stable during development phase

• One or Multi developers

• Minor integration problems

• Long term maintenance

Multi technology integrated applications

SQL Data resources

Multi technology integrated applications

• FME is one part of the codebase (.NET, java, sql, python)

• Dependencies between code of different technologies • DB schema, stored procedures

• Web Services

• NON FME components change every day • You cannot wait for all resources to become stable before you

start developing in FME

• Many integration problems

Multi technology integrated applications

All individual developments must be integrated

before it can be tested

The full application must be build to know if it

works as expected

What is

Continuous Integration?

Continuous integration (CI) No matter how good the individual

code is, the full application will

not work unless it fully integrates

all components and technologies!

Continuous Integration aims to

integrate all code developments

and build the end user application

at least once a day!

CI/CD Key components 1. All requirements and issues tracked in a single incident

tracking system

2. Multi-environment strategy

3. Source Code Version control in a single repository • All technologies using the same build number

4. Automated build • Construct deployment artifact: Compilation or zip

5. Automated testing • Sample static datasets, stops the build in failure

6. Automated deployment • Deployment becomes a formality

Continuous integration

with FME

1. Centralised requirements/incident mgmt

• Document all requirements in a single system

• During initial analysis, design, implementation,

maintenance phase

• Use the requirements and incident identifiers to

comment your source code and tag the different

versions (using plugins where available)

• Example of tools: JIRA, TFS, Redmine, Mantis, ….

2. Multi environment DTAP

• DTAP stands for Development, Testing, Acceptance and Production

• Each environment has different purposes

• Externalize configuration (xml, ini) • Connection parameters

• Data folders

• Read by controller

• Use FME server parameters (set local variables)

3. Version controlling Workspaces • Use a general central source version control system like SVN,

CVS, TFS, GIT, ….

• FMW should be treated as binary

• No merge, no change comparison

• Editors should use a locking mechanism to avoid conflicts

• Check-in must document the Requirements/incident tracker

references

4. Automated build

• Done by the build server

• Zip all FME sources

• Tag the unique build number to every file

.xml <!--build 20160315 -->

.sql --build 20160315

.py # build 20160315

.fmw #! HISTORY="build 20160315"

5. Automated testing

• Make a stable and permanent sample

dataset which allow transformation

verification

• Make an independent workspace to check

and validate the resulting data

• Block the build if problems occur

6. Automated deployment • Deployment on server = Publication

• Use FME server REST API to automate the publication of each FME resource: workspace, resources, Web Services, Topics, …

• The Publication workspace is function of the specific FME project and must be maintained as the FME code evolves

• No publication should ever be done through the workbench once the publication has been automated

• Take into account the different environments (DTAP : Development, Testing, Acceptance and Production)

Required tools

1. A version control system • Git, svn, cvs, TFS, VSS

2. A build server • Jenkins, Bamboo, TFS, …

3. An artifact repository • Nexus, file repository, docker repository

4. A requirements and incident mgmt system • Mantis, JIRA, redmine, TFS

Benefits

• Continuous releasable solution

• Continuously testable solution

• Deployment and publication become a routine stress

less activity

• FME developers can relate bug documentation to

solution build numbers and to FME server workspaces

• Align release management of FME and other code

Conclusion

Version Requirements DTAP Auto

Build

Auto

Testing

Auto

Deploy

Local X X

FME server X X X

Multi

techno

X X X X X X

Conclusion

No more deploy and hope for miracles!

Thank you!