Embed Size (px)
Citation preview
CI Software experiences with Apache Airavata
Suresh Marru, Marlon PierceSudhakar Pamidighantam, Chathuri Wimalasena
Research Technologies
Pervasive Technology Institute/UITS, Indiana University
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Outline
1. Motivation - SciGaP• Operational Sustainability of Science
Gateways
2. Reality vs Fantasy with emerging trends• Micro-Services based architecture, DevOps,
Cloud Native, Big Data Tools…..
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Acknowledgement: SciGaP project funded by NSF ACI Awards:
1339774, 1339856, and 1339649
PI: Marlon PierceCo-I: Suresh Marru
PI: Mark MillerCo-I: Amit Majumdar
PI: Borries Demeler
GridChem Science Gateway
CIPRES Science Gateway
Ultrascan Science Gateway
IU Cyberinfrastructure Gateway
Neuroscience Gateway
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Knowledge and Expertise
Computational Resources
Scientific Instruments
Algorithms and Models
Archived Data and Metadata
Advanced Science Tools
Science Gateways: Enabling & Democratizing Scientific Research
Scale number of gateways without having to scale FTE’s needed to support them
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Science Gateway Platform as a Service (SciGaP)
User Identity Management
Information, Monitoring & Auditing
Application Programmer Interface
CIPRES
Science Gateways
Neuro Science Ultrascan BioVLABGAAMPDES
SimWGParamChem
Graphical Interfaces Admin Dashboards
XSEDE OSGFuture
GridData Nets
Campus Clusters
Academic & Commercial
Clouds
International Grids
Data & Provenance Management
Scalable SecureLoad Balanced Configurable Fault Tolerant Maintainable Performance
Job & Workflow Management
Improve sustainability by converging on a single set of hosted infrastructure services
Meritocratic open governance
Apache Way
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Airavata PMC
• Aleksander Slominski, IBM• Amila Jayasekara, IU• Ate Douma, One Hippo• Chathura Herath, Knight
Capital• Chathuri Wimalasena, IU• Chris A. Mattmann, NASA JPL• Eran Chinthaka, Comprehend• Heshan Suriyaarachchi, • Lahiru Gunathilake, HighFive• Marlon Pierce, IU
• Milinda Pathirage, IU• Patanachai Tangchaisin, Ipsy• Raminder Singh, IU• Saminda Wijeratne, Georgia
Tech• Shahani Weerawarana,
Auxenta• Shameera Rathnayaka, IU• Srinath Perera, WSO2• Supun Nakandala, UoM• Suresh Marru (Chair), IU• Thilina Gunarathne, KPMG
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Apache Community:Extend Airavata from your project or
extend your project from Airavata
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Plans for Continuous Integration & Deployment
• Use Ansible to setup a Apache Mesos Cluster• To be deployed on Quarry Gateway Hosting at IU,
Amazon EC2, Rackspace and in future on JetStream
• Dockerize all Airavata and dependent services• Use Marathon to deploy and maintain docker
containers on the mesos cluster.
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Airavata: A simple and open architecture to facilitate external
contributions
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
• Modern distributed applications are rarely composed of modules written in a single language.
• Weaving together innovations made in a range of languages is a core competency of successful enterprises.
• Cross language communications are a necessity, not a luxury.
Gateways are Polyglot
* source: The Programmer’s Guide to Apache Thrift.
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Apache Airavata API’s built over Apache Thrift
Clean way to define IDLs with richer data structures
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
API’s and CPI’s
• External clients interact with Airavata API (based on Apache Thrift).
• Internally, components interact with each other through Component Programming Interfaces (CPIs).
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Experiences
• Complex use cases evolve over time which challenges the need to preserve a cleaner design.
• Support native clients in various languages.• Thrift implementation of API/CPI services provide robust and scalable
servicers. Different implementations to switch based on QoS (TSimpleServer, TThreadPoolServer TNonblockingServer, THsHaServer).
• No need to marshall / unmarshall data – objects generated from models are used internally as well.
• Auto-generation of Client-SDK’s makes it easy to do modifications to the models.
• Convenient way to achieve backward compatibility.
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
What we learn from fellow Apache Projects:
• Working collaboratively.• Constructive criticism is a
blessing. • Commit early and commit often.• Test driven development
https://cwiki.apache.org/confluence/display/AIRAVATA/Tests+in+Airavata
• Don’t be afraid to make a change, after all version control systems (git, svn) can be trusted.
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Community helps find New Stakeholders
• Current stakeholders have been slow to adopt cloud computing.
• Scientific “big data” could change this.• And we want Airavata to move beyond
academic computing.• Where does Airavata fit in the Apache
stack?
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Architecture Feedback: [email protected]
Not so fast, My friend
Apache Airavata Software ==> Science Gateways Platform as a Service (SciGaP) Hosted Services
Community Hangout
Mailing lists:• [email protected]• [email protected]• [email protected]
Direct Contact: Suresh Marru ([email protected])
?? Questions ??
