© 2012 ADAPTIVE COMPUTING, INC. 1
Building an Integration Platform with Grails and Gradle
Brian Saville
Adaptive Computing
18 Oct 2012
© 2012 ADAPTIVE COMPUTING, INC. 2 © 2012 ADAPTIVE COMPUTING, INC. 2
Me
Brian Saville
Twitter: @bksaville
Plugins:
▪ External Config Reload
▪ Spring Security OAuth2 Provider
▪ New Doc
▪ Artefact Messaging
▪ Contributor on Others
© 2012 ADAPTIVE COMPUTING, INC. 3 © 2012 ADAPTIVE COMPUTING, INC. 3
Adaptive Computing Highlights
▪ Leaders in cloud and HPC software
▪ 10+ years developing Moab® technology
▪ Leadership team w/ $1.5B+ in exits
▪ 15 issued patents, 34 pending
▪ Backed by top-tier investors:
▪ Global Partnerships: HP, IBM, SGI, Cray, Microsoft, etc.
▪ Over 200 customers such as:
© 2012 ADAPTIVE COMPUTING, INC. 4 © 2012 ADAPTIVE COMPUTING, INC. 4
Moab Web Services
▪ What is it?
▪ Grails 2.0.3
▪ RESTful Web Services
▪ Business Logic
▪ Plugin Framework (Beta)
▪ What does it do?
▪ Wraps C Core Product
▪ Integrates “old world” non-standardized API with “new world” APIs
© 2012 ADAPTIVE COMPUTING, INC. 5 © 2012 ADAPTIVE COMPUTING, INC. 5
Atypical Grails Deployment
▪ MWS Deployed on Client Servers
▪ WAR file
▪ Tomcat 6
▪ Oracle/Sun JDK 6
▪ MongoDB 2.0+
▪ Customization on Client Sites
© 2012 ADAPTIVE COMPUTING, INC. 6 © 2012 ADAPTIVE COMPUTING, INC. 6
Integration Platform
▪ Reasons
▪ Extensibility
▪ In-House Developers
▪ External Developers
▪ Dynamic Updates to Application Behavior
▪ Why Create Our Own?
▪ Consists of Plugin Framework (Grails) and SDK (Gradle)
© 2012 ADAPTIVE COMPUTING, INC. 7 © 2012 ADAPTIVE COMPUTING, INC. 7
© 2012 ADAPTIVE COMPUTING, INC. 8 © 2012 ADAPTIVE COMPUTING, INC. 8
Why Do We Care?
▪ Integration Platform Developers
1) Provides a standard pattern to follow
2) Identifies required and optional framework components
3) Treads ground for new integration platforms
4) Identifies hurdles and potential problems
▪ Grails/Gradle Users
1) New advanced application of Grails and Gradle
2) Grails plugin framework can be evaluated with the pattern
3) Gradle plugin framework can be evaluated with the pattern
© 2012 ADAPTIVE COMPUTING, INC. 9
Integration Platform Pattern
© 2012 ADAPTIVE COMPUTING, INC. 10 © 2012 ADAPTIVE COMPUTING, INC. 10
The Pattern
▪ Plugin Identification
▪ Lifecycle and State Management
▪ Dependency Management
▪ Extend Application
▪ Services
▪ Configuration Management
▪ Lifecycle Hooks
▪ Metadata
▪ Development (SDK)
© 2012 ADAPTIVE COMPUTING, INC. 11 © 2012 ADAPTIVE COMPUTING, INC. 11
Plugin Identification
▪ By Convention
▪ Class names
▪ Packaging (JAR, Java or Groovy file)
▪ By Interface(s)
▪ IPlugin
▪ Activator (OSGI)
▪ By Configuration
▪ MANIFEST.MF
© 2012 ADAPTIVE COMPUTING, INC. 12 © 2012 ADAPTIVE COMPUTING, INC. 12
Lifecycle and State Management
▪ Concept of State
▪ Stopped and started
▪ More states optional
▪ Control Over State
▪ How to stop, how to start?
▪ How to add plugin? (Removal not required)
© 2012 ADAPTIVE COMPUTING, INC. 13 © 2012 ADAPTIVE COMPUTING, INC. 13
Dependency Management
▪ Internal Dependencies
▪ Build process
▪ Consider IDE
▪ External Dependencies
▪ Fat JARs
▪ Dependency resolution when adding
© 2012 ADAPTIVE COMPUTING, INC. 14 © 2012 ADAPTIVE COMPUTING, INC. 14
Extend Application
▪ Different in Each Case
▪ Custom Services
▪ Web services
▪ Internal services
▪ Provide Resources
▪ Report or Persist Data
▪ …
© 2012 ADAPTIVE COMPUTING, INC. 15 © 2012 ADAPTIVE COMPUTING, INC. 15
Services
▪ Stateless Services
▪ Control services
▪ Logging
▪ Dependent Services
▪ Configuration
▪ Data persistence
© 2012 ADAPTIVE COMPUTING, INC. 16 © 2012 ADAPTIVE COMPUTING, INC. 16
Configuration Management
▪ Access Application Configuration
▪ (Optional) Access Private Configuration
▪ Preferences
© 2012 ADAPTIVE COMPUTING, INC. 17 © 2012 ADAPTIVE COMPUTING, INC. 17
Lifecycle Hooks
▪ State Transition Hooks
▪ Starting and stopping
▪ Before, after, and/or during transition?
▪ Other Hooks
▪ Setup and configuration
▪ How Are They Defined?
▪ Convention
▪ Interfaces (Configuration)
© 2012 ADAPTIVE COMPUTING, INC. 18 © 2012 ADAPTIVE COMPUTING, INC. 18
Development (SDK)
▪ Dependency Management
▪ Build with internal dependencies
▪ Include external dependencies
▪ Testing
▪ Unit
▪ Integration
▪ Deployment
▪ (Optional) RAD Tool
© 2012 ADAPTIVE COMPUTING, INC. 19 © 2012 ADAPTIVE COMPUTING, INC. 19
Metadata
▪ By Convention
▪ By Configuration
▪ Types of Metadata
▪ Project/Plugin information
▪ Author information
▪ (Optional) Configuration constraints
▪ (Optional) Dependency information
© 2012 ADAPTIVE COMPUTING, INC. 20 © 2012 ADAPTIVE COMPUTING, INC. 20
Non-Essential Pattern Elements
▪ Security
▪ Deployment System/Central Repository
▪ Inversion of Control (IoC) Container
▪ Data Persistence
▪ Individual to Plugin
▪ Application Aware
▪ Common Clients and Parsers
▪ XML/JSON
© 2012 ADAPTIVE COMPUTING, INC. 21 © 2012 ADAPTIVE COMPUTING, INC. 21
Non-Essential Pattern Elements Cont.
▪ Events – Built-In and Custom
▪ Decomposition of Logic
▪ Messaging (i18n)
▪ Startup/Shutdown Concerns
▪ Scheduled Jobs
▪ Inter-Project Dependencies
▪ Environment Handling
© 2012 ADAPTIVE COMPUTING, INC. 22 © 2012 ADAPTIVE COMPUTING, INC. 22
Grails Plugins
▪ What is Missing?
▪ Standardized configuration management
▪ Platform core is addressing this
▪ Framework is Mature
© 2012 ADAPTIVE COMPUTING, INC. 23
MWS Integration Platform
© 2012 ADAPTIVE COMPUTING, INC. 24 © 2012 ADAPTIVE COMPUTING, INC. 24
Introduction
▪ Plugin Type – Plugin Definition Class
▪ One per Class
▪ Stored as Groovy or JAR files
▪ Internally is a Grails artefact
▪ Plugin Instance
▪ Multiple instances for a single plugin type possible
▪ Internally is DB entry and Spring Bean
▪ Dynamic (Running Application)
© 2012 ADAPTIVE COMPUTING, INC. 25 © 2012 ADAPTIVE COMPUTING, INC. 25
Plugin Identification
▪ “Ends With” Class Names
▪ Plugin Types
▪ Translators
▪ Autowired Classes
▪ Spring Annotated Classes
▪ @Component
▪ @Autowired
▪ @Resource
▪ @Scope
© 2012 ADAPTIVE COMPUTING, INC. 26 © 2012 ADAPTIVE COMPUTING, INC. 26
Lifecycle and State Management
▪ States
▪ Started, Stopped, Paused
▪ Plugin Control Service
▪ Start, stop, pause, resume
▪ Adding Plugins
▪ User Interface
▪ REST API
© 2012 ADAPTIVE COMPUTING, INC. 27 © 2012 ADAPTIVE COMPUTING, INC. 27
Dependency Management
▪ SDK for Build Dependencies
▪ Grails JARs
▪ Spring JARs
▪ Support Fat JARs
▪ Dependencies Resolved Before Adding
© 2012 ADAPTIVE COMPUTING, INC. 28 © 2012 ADAPTIVE COMPUTING, INC. 28
Extend Application
▪ Reporting – Application Aware Data Persistence
▪ Expose Custom Web Services
▪ Methods by convention
▪ Almost full REST API
▪ Unsecured annotation
▪ Routed by URL mappings
▪ Built-In Events by Convention
© 2012 ADAPTIVE COMPUTING, INC. 29 © 2012 ADAPTIVE COMPUTING, INC. 29
Services
▪ Logging (Similar to Grails)
▪ Data Persistence Services
▪ Control Service
▪ Individual Datastore Service
▪ SSL Service - SSLSocketFactory
▪ not-yet-commons-ssl with PEM files
▪ In memory keystore
▪ Internal REST Service
▪ Request and response with optional attributes
▪ Returns object with response, easy access properties, parsed data
© 2012 ADAPTIVE COMPUTING, INC. 30 © 2012 ADAPTIVE COMPUTING, INC. 30
Configuration Management
▪ Application Configuration
▪ Grails application bean
▪ Plugin Private Configuration
▪ Metaclass injected “config” property
© 2012 ADAPTIVE COMPUTING, INC. 31 © 2012 ADAPTIVE COMPUTING, INC. 31
Lifecycle Hooks
▪ Implemented as Methods on Plugin Types
▪ beforeStart, afterStart, beforeStop, afterStop
▪ configure
▪ If Not Implemented, Inject Empty Methods
▪ Called from Control Service
© 2012 ADAPTIVE COMPUTING, INC. 32 © 2012 ADAPTIVE COMPUTING, INC. 32
Metadata
▪ Convention
▪ Plugin project compiled class included in JAR
▪ Ends with “Project”
▪ Configuration
▪ Properties on plugin project class
▪ =~ Grails Plugin Class
© 2012 ADAPTIVE COMPUTING, INC. 33 © 2012 ADAPTIVE COMPUTING, INC. 33
Metadata – Dynamic Configuration Constraints
▪ Configuration is Dynamic Map
▪ Utilize Grails Constraints and Conventions
▪ Redo min/max to remove value checking
▪ Add custom plugin constrained property (CP)
▪ Add custom plugin CP builder
▪ Add Custom Constraints
▪ Type (Class) – Infer if not specified
▪ Default value
▪ Required (nullable)
▪ Scriptable URL (exec, file, http)
▪ Customized validator with additional methods
© 2012 ADAPTIVE COMPUTING, INC. 34 © 2012 ADAPTIVE COMPUTING, INC. 34
i18n Messaging
▪ Message Property Files
▪ Included in JAR
▪ Loaded with individual message source per project
▪ Message Method Injected on Plugins
▪ Used to Resolve Constraint Errors
▪ Not quite multi-lingual
© 2012 ADAPTIVE COMPUTING, INC. 35 © 2012 ADAPTIVE COMPUTING, INC. 35
Development (SDK)
▪ Commons JAR
▪ Interfaces, classes, constants, annotations
▪ Testing JAR – TestFor
▪ Gradle Project
▪ Dependencies
▪ Testing and Commons
▪ Creates fat JARs (lib and resolved deps)
▪ Additional build tasks
▪ Handles i18n property files
▪ Cobertura code coverage reports
▪ Unit testing with Spock
© 2012 ADAPTIVE COMPUTING, INC. 36 © 2012 ADAPTIVE COMPUTING, INC. 36
Development (SDK) - Commands
▪ Gradle
▪ create-project -Pproject-name=MyProject
▪ create-plugin -Pplugin-name=sample.MyPlugin
▪ test
▪ upload-test -Pmws.url=http://localhost/mws
▪ (Eventually) Python CLI Wrapper
▪ mplugin create-project MyProject
▪ mplugin create-plugin sample.MyPlugin
▪ mplugin test
▪ mplugin upload --url=http://localhost:8080/mws
© 2012 ADAPTIVE COMPUTING, INC. 37 © 2012 ADAPTIVE COMPUTING, INC. 37
Non-Essential Pattern Elements Status
▪ Security - Yes
▪ Central Repository – No
▪ IoC Container – Spring
▪ Data Persistence – Individual Datastore and Application Aware
▪ Common Clients and Parsers – Client Decided and Groovy
▪ Events – Built-In Only
▪ Decomposition – IoC with Conventions and Annotations
© 2012 ADAPTIVE COMPUTING, INC. 38 © 2012 ADAPTIVE COMPUTING, INC. 38
Non-Essential Pattern Elements Status Cont.
▪ Startup/Shutdown Concerns – Initial Plugins
▪ Scheduled Jobs – Quartz
▪ Inter-Project Dependencies – No
▪ Environment Handling – Application Configuration Parameter
© 2012 ADAPTIVE COMPUTING, INC. 39 © 2012 ADAPTIVE COMPUTING, INC. 39
Comparison to Grails Plugins
Concept MWS Plugins Grails Plugins
RAD Tool Partial X
Configuration X No Standard
Integration Tests X
Multiple Instances
X
Custom REST Partial X
Metadata Partial X
Languages Groovy Groovy & Java
Dynamic X
Lifecycle X X
i18n Partial X
© 2012 ADAPTIVE COMPUTING, INC. 40 © 2012 ADAPTIVE COMPUTING, INC. 40
Demo
© 2012 ADAPTIVE COMPUTING, INC. 41
Problems Encountered
© 2012 ADAPTIVE COMPUTING, INC. 42 © 2012 ADAPTIVE COMPUTING, INC. 42
JVM ClassLoaders
▪ Separating ClassLoaders
▪ Locking JARs
▪ Fat JARs
▪ Unresolved: Reloading Typed References
© 2012 ADAPTIVE COMPUTING, INC. 43 © 2012 ADAPTIVE COMPUTING, INC. 43
Language Support
▪ Groovy – Works Great
▪ Java – Interfaces and Metaclass Issues
© 2012 ADAPTIVE COMPUTING, INC. 44
Questions?