Developing Computational Science Gateways using Apache Airavata

Marlon PierceScience Gateway Group, Indiana University

marpierc@iu.edu

1. We Are Hiring

Apply: http://goo.gl/LbiXNO

http://www.catonmat.net/blog/what-would-john-mccarthy-say-if-he-saw-you-programming/

2. dev-subscribe@airavata.apache.org

3. We’ll get to this one later….

http://www.bayoubelles.com/2011/08/night-before-big-day-was-not-what-was.html

Airavata’s Philosophy• There a lots of ways to build Web interfaces for

Science Gateways.– By Hand: PHP, Twitter Bootstrap, AngularJS, …– Turnkey Frameworks: Liferay, Drupal, Plone, Joomla, …– Science Gateway Frameworks: the SDSC Workbench,

HUBzero, gUse/WS-PGrade• Gateway developers should concentrate on

building interfaces that serve their community.• And outsource the general purpose services to

Airavata.

Airavata Role and Goals: Improve sustainability by converging on a single set of hosted infrastructure services

Apache Airavata ComponentsComponent Description

Airavata API Server

Apache Thrift-generated server skeletons of the API and data models; directs traffic to appropriate components

Registry Insert and access application, host machine, workflow, and provenance data.

Orchestrator Handles experiment execution request validation, scheduling, and decision making; selects GFAC instances that can fulfill a given request

GFAC Manages the execution and monitoring of an individual application.

Workflow Interpreter

Execute the workflow on one or more resources.

Messaging System

WS-Notification and WS-Eventing compliant publish/subscribe messaging system for workflow events

Airavata Components

Airavata’s Future Directions: SciGaP

• SciGaP: Airavata as a multi-tenanted Gateway Platform as a Service

• Goal: We run Airavata so you don’t have to.– Scalable support

• Challenges: –Centralize system state–Make Airavata more cloud friendly, elastic

http://scigap.org

Some Contribution OpportunitiesComponent Research Opportunities

Registry Better support for Thrift-generated objects; NoSQL and other backend data stores; fault tolerance

Orchestrator Pluggable scheduling; load balancing and elasticity

GFAC ZooKeeper-like strategies for configuring and managing.

Messenger Investigate AMQP, Kafka, and other newer messaging systems

Workflow Interpreter

Alternative workflow processing engines.

Overall Message-based rather than direct CPI calls.

Airavata components expose Component Programming Interfaces (CPIs) that allow you to switch out implementations. GFAC is also designed to be

pluggable.

Airavata and the Apache Software Foundation

Experiences with Open Governance

Cyberinfrastructure: How open is open source?

• What’s missing?– Open source licensing– Open Standards– Open Code (GitHub,

SourceForge, Google Code, etc)

We also need Open Governance

Open Source Software and Governance

• Open source projects need diversity, governance.– Sustainability

• Incentives for projects to diversify their developer base.

• Govern– Software releases– Contributions– Credit sharing.– Members are added– Project direction decisions.– IP, legal issues

• Our approach: Apache Software Foundation

Collaborate

Compete

“Apache” Means “Open”

Join the Airavata developer or architecture mailing lists, get involved, submit patches, contribute.

Use Give Back

More Information

• Contact Us:– marpierc@iu.edu, smarru@iu.edu– Join dev@airavata.apache.org,

users@airavata.apache.org, architecture@airavata.apache.org

• Websites:– Apache Airavata: http://airavata.apache.org – SciGaP Project: http://scigap.org – Science Gateway Institute: http://sciencegateways.org

Airavata AudiencesAudience Description

Gateway Developers

Use the Airavata API through and SDK in their favorite programming language.

Airavata Developers

Want to change Airavata components, experiment with different implementations.

Middleware Developers

Want to extend Airavata to talk to their middleware clients.

Resource Providers Want to configure Airavata to work with their middleware.

Apache Contributions Aren’t Just Software

• Apache committers and PMC members aren’t just code writers.

• Successful communities also include– Important users– Project evangelists – Content providers: documentation, tutorials– Testers, requirements providers, architects, and

constructive complainers• Using Jira and mailing lists

– Anything else that needs doing.

Getting Involved, Contributing Back

• Airavata is open source, open community software.

• Open Community: you can contribute back– Patches, suggestions, wiki

documentation, etc• We reward contributors

– Committers: write access to master Git repo

– Project Management Committee members: full, binding voting rights

Airavata AudiencesAudience Description

Gateway Developers

Use the Airavata API through and SDK in their favorite programming language.

Airavata Developers

Want to change Airavata components, experiment with different implementations.

Middleware Developers

Want to extend Airavata to talk to their middleware clients.

Resource Providers Want to configure Airavata to work with their middleware.

The Apache Software Foundation

• Apache software powers 65% of web sites worldwide

• 501(c)3 non-profit foundation

• Reasons for creating ASF– Create legal entity– Protect contributors from

liability– Protect Apache assets

• Membership: individual• Apache Incubator

• Governance and Staffing– Board of Directors– Project Management

Committees– ASF Members– Committers– Contributors

• Funding– All-volunteer

staffing/development resources

– Donations– Corporate investment

Where Is Airavata 1.0?

• Airavata 1.0 will be the stable version of the API.

• Version 1.0 determined by developer community vote.

• Semantic versioning

Airavata Thrift Definitions

airavata/airavata-api/thrift-interface-descriptions

Thrift File Contents

airavataAPI Definitions of services; top level file for other Thrift files.

airavataErrors Definitions of exceptions thrown by the API services.

airavataDataModel Top level container for data models.

workspaceModel Definitions of the structs (Project, User, Gateway, Group); includes experimentModel

experimentModel Definitions of data structures that constitute the experimentModel.

applicationCatalogAPI App catalog service definitions.

applicationCatalogDataModel App catalog data model definitions.

Airavata API and Apache Thrift

• We use Apache Thrift to define the API.• Advantages of Thrift– Supports well-defined, typed messages.– Custom defined errors, exceptions– Generators for many different programming languages.– Some shielding from API versioning problems.

• Downsides of Thrift– No message headers, so everything must be explicitly

defined in the API.

A Few Observations on Successful Gateways

• Support familiar community applications.• Make HPC systems easy for new user

communities who need HPC.• Keep it simple.• Have champions who build and support the

community.• Have a lot of common features.

Airavata API and Apache Thrift

• We use Apache Thrift to define the API.– TCP/IP typically– Not WS, not REST

• Advantages of Thrift– Supports well-defined, typed messages.– Custom defined errors, exceptions– Generators for many different programming languages.– Some shielding from API versioning problems.

• Downsides of Thrift– No message headers, so everything must be explicitly defined in

the API.

Airavata API Thrift Definitions

airavata/airavata-api/thrift-interface-descriptions

Thrift File Contents

airavataAPI Definitions of services; top level file for other Thrift files.

airavataErrors Definitions of exceptions thrown by the API services.

airavataDataModel Top level container for data models.

workspaceModel Definitions of the structs (Project, User, Gateway, Group); includes experimentModel

experimentModel Definitions of data structures that constitute the experimentModel.

applicationCatalogAPI App catalog service definitions.

applicationCatalogDataModel App catalog data model definitions.