Open Grid Computing Environments TutorialMarlon Pierce, Suresh Marru, Gopi Kandaswamy, Gregor von Laszewski, and Tom Scavo

Tutorial OverviewPart 1: OGCE Portal (Marlon)o Introduction and overviewo Portal Demoo How to build Grid portlets and gadgets

Part 2: OGCE Workflow Tools (Suresh and Gopi)o Part A: Demo and overviewo Part B: Hands on demo, deploying new

applications

Part 3: Web 2.0 for Grid Portals (Gregor)

Part 4: GridShib and Gateway security (Tom)

Goals of TutorialGoalso Review portal and gateway technologies old and new.o Demonstrate how to execute workflows and wrap

TeraGrid applications.o Discuss security o Hear from TG Gateway community: what do you

want?

We have a lot of material to cover so we will sample many things.o You will have enough information to get going as a

homework assignment.o Please also check with us at TG08

Please see Nancy’s “Gateway Parade” this afternoon.

Slides and Demo SiteTutorial slides are available from http://www.collab-

ogce.org

We run a permanent demo portal at https://community.ucs.indiana.edu:8443/gridsphere/o Also aliased as

https://ogceportal.iu.teragrid.org:8443/gridsphere

Demo services are also available from Communityo [NEED URLS]

Accounts train30-train49 have been created for the workshop. Password is the same as the account name.

We also have training accounts with same names that can be used to retrieve TG proxy credentials.

Portal and Gadget DemosDemonstration of basic capabilities

Getting the Code

What You Can DownloadOGCE Portalo Various portlet samplers

Web and Grid Serviceso XRegistry: Web Service

Registryo GFAC: Application Factory

Serviceo Grid Portal Information

Repository

Programming Librarieso Java and JavaScript COGo Grid Tag Libraries and

Beans (GTLAB)o Information Service Client

APIs

Project Release ScheduleWe have two major releases per year with

intermediate updates.o TeraGrid (June)o Supercomputing (November)

Focus of this release cycle: the workflow suite tools shown in Part 2.

We do not currently package services and portlets together. o These are separate downloads.

Project Build and TestSource Code Version Control

and Building: We use Maven 2 and SourceForge’s SVNo Anonymous checkoutso We are Open Source

Nightly Build Testing: We use the NMI Build and Test facility at UWo Mac and Linux flavors

System Testing: We use Apache JMeter to verify portal builds function correctly.o You can download this from

uso You can also view the nightly

dashboard

Project InformationWeb site: http://www.collab-ogce.org

Announcement Blog: http://collab-ogce.blogspot.com

RSS/Atom Feeds: http://collab-ogce.blogspot.com/atom.xml oThis is also embedded in the wiki as

“News”

Contact us: discuss@ogce.org

Building the Portal

Choosing a Host ComputerSelect your host computer.o You will need a real IP addresso VPN also OK (?)o Know general Globus et al firewall issues.

I recommend turning firewalls and SELinux off for development.

Operating Systemso We test with Mac OS X and many Linux flavorso Other Unix should be fine (assuming Java works)o We have not had any requests for Windows.

Software Environmento The portal download comes with everything you need

EXCEPT Java. You need the JDK, not the JRE.o We test with Java 1.5. Don’t use Java 1.4.

The more memory, the better.

Downloading and Building the PortalDownload the portal from

http://www.collab-ogce.org.o Unpack in $HOME.

You should also download the Maven repository.o This is optional but

recommended

Edit ogce-portal-home/pom.xml

Run the command “mvn clean install”.o Use “mvn –o clean install”

to build offline.

Build will typically take 5 minutes (new Mac) to 30+ minutes (old Linux PC).

You can also check out the latest code from SVN. Check the “Nightly Build” dashboard to verify.

Practical ConsiderationsIncrease your JVM Memoryo export JAVA_OPTS="-server -Xms512m -

Xmx1024m -XX:MaxPermSize=256m”o Xmx2048m, 4096m, etc if you can.

Monitor catalina.out for problemso tail –f catalina.out

Make sure that Tomcat shuts downo Kill if necessary

NFS can really slow down buildingo Use /usr/local/ if you have problems

Editing pom.xmlYou only need to edit the

top level ogce-portal-only/pom.xml.

You only need to change <properties> for local settings.

You MUST update the IP address.

You may want to add or remove GRAM and GridFTP hosts.

Update project.home if you are building someplace besides $HOME.

<properties> <portal.server.ip> 156.56.104.143 </portal.server.ip>

<host.base.url> http://${portal.server.ip}:8080/ </host.base.url> <gridftp.host.names> …. </gridftp.host.names> <gram.host.names> …. </gram.host.names>

<project.home> ${env.HOME}/ogce-portal-only </project.home>

</properties>

Rebuilding the PortalRunning “mvn clean install” will completely

wipe out the existing installation.o You may not want to do this.

If you want to rebuild individual components, just run the following command from the ogce-portal-only directory.o mvn clean install –f

portlets/myportlet/pom.xml

You can do this for all <modules> in ogce-portal-deploy/pom.xml

OGCE Portal Project LayoutWe follow standard Maven 2

conventions for project layouts for each component.omoduleName/src/main/webappomoduleName/src/main/java

Modules are listed on next slideoSee ogce-portal-only/pom.xml. oEach module is a separate directory

and has its own pom.xml.

Module Description

Portal_deploy This is where your portlets are deployed into Tomcat.

Containers Specific build instructions for containers. Currently we only have GridSphere 2.1.5 here.

Lnf_portal Controls look and feel for the container.

Global-config Deploys COG jars into /shared/lib and applies Tomcat custom configs.

Certificates CA certs. Includes download of all TG certs.

Portlets Each portlet has its own module.

Developing Grid User Interface ComponentsPortlets, gadgets, etc.

Under the Hood: Building ComponentsOGCE portlets use the Java COG Abstraction Layer to

access Grid services.o Provides a buffer over different Grid toolkits and

versions.o Provides ability to compose graph-based workflows.o Tutorial in itself

Portlets can be built with our Apache Velocity bridge.o “Template” or “steal this code” approach.

We have also developed Grid Tag Libraries and Beans (GTLAB) to simplify this process.

GTLAB FeaturesExtends Java Server Faces.o Tag components wrap major COG Abstraction

Layer features

Allows you to do standalone development.

Use JSF portlet bridge to convert into portlets No new coding, just change, add config files and jars.

Or forget about portlets and o Convert into Google Gadgetso Develop as a Facebook applicationo Etc.

Installing GTLABYou need to install the OGCE portal o GTLAB depends on many shared jars.

Download the release code from www.collab-ogce.org or check out from SVN:o svn checkout

https://ogce.svn.sourceforge.net/svnroot/ogce/GTLAB

Edit GTLAB/jsf_standalone/pom.xml o <catalina.home> should point to your OGCE portal server.

Edit GTLAB/jsf_standalone/src/main/resources/resources.properties.

Run “mvn clean install” in GTLAB/jsf_standalone/

Levels of Abstraction in GTLAB code

You can use prepackaged gadgets/portlets.

You can mix and match repeatable sections of tags via “include” statements.

You can develop new applications using existing tag libraries.

You can hack the codeNote JSF is compatible with JSP, so you can

mix and match.

And you can embed JavaScripto YUI, Scriptaculous user interface goodies.o Google and Facebook APIs.

MyProxy Example<h:form id="form"> <h:outputText value="Username:"/> <h:inputText value="#{resource.username}" required="true"/>

<h:outputText value="Password: "/> <h:inputSecret value="#{resource.password}" required="true"/>

<h:outputText value="MyProxy Server: "/> <h:inputText value="#{resource.myproxyHostname}" required="true"/>

<o:submit id="submit" value="Submit" binding="#{builder.body}" action="#{builder.junkAction}" >

<o:multitask id="multi" persistent="true" taskname="#{resource.taskname}"> <o:myproxy id="mypr" hostname="#{resource.myproxyHostname}"

lifetime="2" password="#{resource.password}" port="7512"

username="#{resource.username}"/> </o:multitask> </o:submit> </h:form>

Show the Proxy Credential<h:dataTable value="#{myproxy.storedProxyList}" border="1"

binding="#{myproxy.storedProxyTable}" var="proxyEntry" id="proxyTable">

<h:column> <f:facet name="header"> <h:outputFormat id="subject" escape="false”

value="Subject"/> </f:facet>

<h:outputText id="proxySubject" value="#{proxyEntry.subject}"/> </h:column>

<h:column> <f:facet name="header"> <h:outputFormat id="Remove" escape="false”

value="Delete"/> </f:facet>

<h:commandButton id="deleteProxy" value="Remove” actionListener="#{myproxy.deleteCredential}"/>

</h:column></h:dataTable>

Executing Multiple Task<o:submit id="submit" value="Submit" binding="#{builder.body}"

action="#{builder.junkAction}" > <o:multitask id="multi" persistent="true"

taskname="#{resource.taskname}"> <o:jobsubmit id="js" myfaces="true"

arguments="#{resource.arguments}" executable="#{resource.executable}" hostname="#{resource.hostname}" provider="#{resource.provider}" stdout="#{resource.stdout}"/>

<o:filetransfer id="ft" myfaces="true" from="#{resource.from}"

to="#{resource.to}" /> <o:dependency id="dep" task="ft"

dependsOn="js"/> </o:multitask></o:submit>

GTLAB Example Description

MyProxyExample Get a credential and store in OGCE global proxy store.

JobSubmitExamples

Various examples showing how to submit a grid job and get output.

Dashboard Example

Runs showq on Big Red and formats output.

DirectoryBrowser Displays and downloads files on a remote host via GridFTP

MultipleTaskExample

Shows how to couple file transfers, operations, and

TaskMonitorExample

View task state and metadata.

These will work as gadgets, portlets, or standalone applications

Making GTLAB Gadgets into PortletsDo all development in jsf_standalone.

When you are ready to convert:o cd GTLAB/transitiono mvn clean process-resourceso cd GTLAB/portletso mvn clean install

You will need to manually edit portlet.xml, group.xml, and layout.xml in GTLAB/portlet/src/main/webapp/WEB-INF/

That’s it.

Making GTLAB Google GadgetsA Gadget is a standalone application that is integrated

into your personal iGoogle display.o They can run on your Web server.o You can use SSL/HTTPS

Google Gadgets have two versionso http: these use Google’s JavaScript APIso url: these wrap external applications with IFrames.

Limitations:o Not much real estate (but location, location,

location) o Cookies manage state on gadgets from the same

server but we need a way to integrate multiple independent servers.

Some GTLAB Gadgets

Example Gadget Config File

<?xml version="1.0" encoding="UTF-8" ?> <Module> <ModulePrefs title="MyProxy Gadget Example"

scrolling="true" height="500"/>

<Content type="url" href="http://…/:8080/GTLAB/examples/

MyProxyExample.jsf"> </Content> </Module>

Save this as MyProxy.xml and place in a Web accessible place.

Some Web 2.0 ResourcesSee recent tutorials by Geoffrey Fox and me:o http://grids.ucs.indiana.edu/ptliupages/

presentations/Web2.0Tutorial.ppto http://grids.ucs.indiana.edu/ptliupages/

presentations/CTSTechFutures_May19-08.pptxo Includes material on Google Gdata, Facebook

API, Open Social, Microformats, etc.

See also full examples at http://communitygrids.blogspot.com

TeraGrid Information Services: Portlets and Gadget LibrariesSlides courtesy of Maytal Dahan

TG Information ServicesWhat are TG Information Services?o Services that publish metadata about

TeraGridd TG Software, TG resources, TG Services

o Software and applications can query TG Info services for latest information

Existing TG Info Services:o Currently we are relying on the REST

interface of TG Info Services TG Also has WebMDS interfaces

o Service includes CTSS4 information

Libraries for TG Info ServicesUsing the REST Interfaces we areo Developing JavaScript Libraries that

automatically query and return service information

o Handling all the querying and xml parsing necessary and return objects through libraries

o API will include the ability to (and more) getTGSites() – Get a list of all TG sites getResoures() – Get list of resources at a site getServices() – Get list of TG resources

Late summer availability

Demo portlets available now

Sample Portlet Interfaces

Talks, Demos, and BOFs 37

Monday 1:00 pm Science Gateways on Parade (Nancy Wilkins-Diehr and a cast of 1000’s)

Tuesday 11:00 am TALK: TeraGrid Open Life Science Gateway (Wenjun Wu)

Tuesday 1:30 pm TALK: Engaging with the LEAD Science Gateway Project: Lessons Learned in Successfully Deploying Complex System Solutions on TeraGrid

Tuesday 3:30 pm TALK: Reliable and Scalable Job Submission: LEAD Science Gateway's Testing and Experiences with WS GRAM on TeraGrid Resources

Tuesday 3:30 pm TALK: The LEAD-WxChallenge Pilot Project: The Potential of Grid-Enabled Learning

Tuesday 5:30 pm BOF: TeraGrid User Portal BOF (Maytal Dahan)

Wednesday 10:30 am TALK: Workflow Infrastructure for Multi-scale Science Gateways

Wednesday 2:00 pm TALK: Increasing TeraGrid User Productivity through Integration of Information and Interactive Services (Maytal Dahan)

Wednesday 4:00 pm TALK: On-Demand Severe Weather Forecasts Using TeraGrid via the LEAD Portal

Wednesday 6:30 pm POSTER/DEMO: TeraGrid Open Life Science Gateway

Wednesday 6:30 pm POSTER/DEMO: Fault Tolerance and Recovery of Scientific Workflows in LEAD (Gopi)