NGS induction --- case study: the BRIDGES project

Micha BayerGrid Services Developer, BRIDGES projectNational e-Science Centre, Glasgow Hub

The BRIDGES project

Biomedical Research Informatics Delivered by Grid-Enabled Services

2 year e-Science project, started 1st October 2003

aim: provide data integration and grid-based compute power for Cardiovascular Functional Genomics project

CFG project investigates genetic predisposition for hypertensive heart disease

my role on project: develop grid applications for end users

BRIDGES requirements and the NGS

functional:

high throughput compute tasks, e.g. large BLAST jobs

non-functional:

interfaces to applications should be targeted at the less computer literate --- users range in computer literacy from fairly advanced to mildly technophobic

security requirements should not cause any extra work or inconvenience for users as this may put them off altogether

resources provided by BRIDGES compete with familiar, similar resources already on offer at established bioinformatics institutions (EBI, NCBI, EMBL) -> need to make things “palatable” so people do use it

How to get your job onto the NGS

NGS clusters

NGS portal

GSI-SSH

project portal

standalone GUI client

custom solutions:

standard solutions:

Leeds

Oxford

RAL

Manchester

Custom grid applications

if possible/appropriate, get a developer to write bespoke interface to a grid app running on NGS

only worthwhile if application is used frequently and/or by many users and is relatively unchanging/simple

best to hide complexity of grid from users altogether

users should not even have to choose between resources

automatic scheduling of jobs to resources that currently have spare capacity is desirable

best option for delivery is portlet in project-specific web portal – just need web browser for access then

Project web portals

portals are configurable, personalized collections of web applications delivered to a web browser as a single page

NGS encourage projects to maintain their own web portals to deliver apps to their users

applications can then be provided through user-friendly, specific portlet interfaces

allows the hiding of grid complexity from users

requires developer time

BRIDGES portal currently uses IBM Websphere (free to academia)

More on portals

increasingly important technology – not just for grid computing (cf. Yahoo)

gives end users a customized view of software and hardware resources specific to their particular application domain

also provides a single point of access to Grid-based resources following user authentication (“single-sign-on”)

content is provided by portlets (Java servlet extension) – JSR168 standard provides for exchangeability

some portal packages currently available: IBM Websphere, Gridsphere, JetSpeed, uPortal, Jportlet, Apache Pluto

Authentication and User Management (1)

model adopted in BRIDGES:

requirement was for users not to have to obtain and manage certificates

we applied for a single project account at NGS – users do not need individual NGS accounts

this account maps to a single user (“BRIDGES”) on the NGS with home directories on all nodes (like normal users)

authentication for this user on NGS is by means of the host certificate of the machine where the jobs are submitted from (under control of BRIDGES project)

users authenticate via the BRIDGES web portal using standard username and password pairs

Authentication and User Management(2)

Users can create accounts for themselves in BRIDGES Websphere portal (“self-care”)

alternatively one could of course give the users usernames and passwords

information gathered is kept in Websphere's secure user database

current info is very basic but will be extended to include more detail (e.g. URL of user's project or departmental website where the user is listed)

provides at least a basic means of accounting for user activity

no need for physically visiting the Registration Authority/presenting ID

may need to resort to stricter security if system is abused e.g. if impersonation takes place etc.

Authorisation with PERMIS

PERMIS = grid authorisation software developed at Salford University (http://sec.isi.salford.ac.uk/permis/)

BRIDGES uses PERMIS to differentially allow users access to resources

typical use is with GT3.3 service but lookup-type use is also possible with other services (in our case GT3.0.2)

code in our service calls a PERMIS authorisation service running on a machine at NeSC

user's roles are queried and access to resource is permitted or denied accordingly

gives BRIDGES staff full control over who is allowed to use NGS resource through our applications

NGS

Leeds

Oxford

RAL

Manchester

end user

ScotGRID

NeSC Condor Pool

Security in BRIDGES – summary

NGS clusters

Leeds

Oxford

RAL

Manchester

end user BRIDGES web portal

NeSC machine with PERMIS authorisation service (GT3.3)

NeSC grid server with host credentials

authenticate at BRIDGES web portal with username and

password only

job request is passed on

securely with username

get user authorisations

make host proxy, authenticate with NGS

and submit job

Host authentication for job submission

allows us to submit jobs to NGS as user “BRIDGES”

apply for host certificate for the grid server machine as normal (UK e-Science Certification Authority)

results in a passwordless private key and host certificate for the machine

Java Cog kit code can then be used to generate a host proxy locally

this is used for job submission

Use case: Microarray reporter sequence BLAST jobs

microarray chips contain up to 400,000 reporter sequences

these need to be compared to existing annotated sequence databases

takes approx. 3 weeks to compute against human genome on average desktop machine

“Job processing – please wait....”(and wait....and wait....)

BLAST

Basic Local Alignment Search Tool

used for comparing biological sequences (DNA, protein) against a set of target sequences

returns a sorted list of matches

most widely used algorithm for this sort of thing

compute intensive

How do I get my application to run efficiently on a grid?

applications to be deployed on a compute grid need to be parallelised to really benefit (can of course just run them as single jobs too)

for this one must be able to partition a job into several subjobs

these then get processed separately at the same time on multiple processors

need to combine results of individual subjobs at the end

Parallel BLAST – grid style

partition your job by putting one or several query sequences into a separate input file (= 1 subjob)

distribute all input files, the executable and target data onto your grid clusters (“stage-in”)

results are returned to the server and combined there

if 100 free processors are available, and 100 subjobs are to be run, the time taken is 1/100th of the time it would have taken to run the whole job on a single machine (plus overheads for scheduling, data transfer and result combining)

To stage or not to stage? file staging is the copying – at runtime – of files onto the remote

resource

example: BLAST jobswe need

input file target data file (“database” – really a flat text file) executable (BLAST)

target files and executable are unchanging components for this kind of job

it is best to store these locally on the remote resources to avoid staging overhead (target data are in the region of several gb in size and growing exponentially)

rather than individual users keeping multiple copies of publicly available data in their home directories, get sys admins to put up copies visible to all

must stage in input files since these vary from job to job

BRIDGES GridBLAST Job Submission

end user machine

NESC Grid Server (Titania)

GT 3 core grid service

return result

send job request

PBS wrapper

BRIDGES Meta-

Scheduler

ScotGRID masternode

ScotGRID worker nodes

jobs farmed out to compute nodes

PBS server side

+ BLAST

NESC Condor pool

Condor Central Manager

Condor+

BLAST

GridBLAST client

Apache Tomcat Condor

wrapper

execution hosts

Oxford headnode

GT2.4+

BLAST

execution hosts

Leeds headnode

GT2.4+

BLAST

GT2.4 wrapper

NGS

Current status of our system

software is still at prototype stage – haven’t benchmarked any really big jobs yet

Java webstart client (launched from portal) connects to service – needs to be changed to portlet

user registration needs to be revised and users re-registered

happy to share portlet code etc with others once finished

How we worked with the NGS

BRIDGES was one of the first projects doing bio stuff on NGS

we established a basic infrastructure needed for BLAST on the NGS clusters in collaboration with NGS user support

good collaboration on our security requirements – very helpful and accommodating

our project account is the first of its kind and we jointly tailored a solution that would fit BRIDGES

ask for what you need! things are not cast in stone and it is supposed to be a public service

Public bioinformatics infrastructure on NGS – current status

we are in the process of establishing an infrastructure for BLAST jobs that can be used by all

this includes:

making BLAST and mpiBLAST executables publicly available

mirroring the entire NCBI BLAST databases repository

currently trialling this on Leeds node – will be replicated at other nodes eventually

data replication on all nodes necessary to avoid severe performance hits

input from others needed and welcome!

Contact details

BRIDGES website: http://www.brc.dcs.gla.ac.uk/projects/bridges/

Code repository (available soon): http://www.brc.dcs.gla.ac.uk/projects/bridges/public/code.htm

BRIDGES web portal: http://europa.nesc.gla.ac.uk:9081/wps/portal

Contacts:

Micha Bayer at NeSC in Glasgow -- michab@dcs.gla.ac.uk

Richard Sinnott at NeSC in Glasgow -- ros@dcs.gla.ac.uk