Embed Size (px)
Citation preview
Outline
• Extended Collaborative Support Services• XSEDE and Science Gateways• The XSEDE portal• XSEDE Resources
2
September 4, 2015
Extended Collaborative Support Services
XSEDE offers huge variety of resources
• Leading‐edge distributed memory systems• Very large shared memory systems• High throughput systems• Visualization engines• Data analytics systems• Accelerators and co‐processorsMany scientific problems have components that call for use of more than one architecture.
4
Extended Collaborative Support Services Strategic Objective -To Help Users Make More Productive Use of XSEDE Resources
• Support people who understand the discipline as well as the systems (perhaps more than one support person working with a project).
• 37 FTEs, spread over ~80 people at almost a dozen sites.
5
How do you get Extended Collaborative Support?• You have to ask for it ‐it’s an allocated resource.
• You can always ask for it, i.e. midstream or even as part of a startup request.
• Lasts up to a year.• Must have specific goal in mind‐ can’t just say I want additional programming support.
6
ECSS has 5 support services
• Support for –Research Teams–Novel and Innovative Projects–Community Capabilities–Gateways– Training and Outreach
7
Support for Research Teams(led by Lonnie Crosby, NICS (U. of Tennessee))• Optimization
– Profiling– Scaling to higher core count– Improving IO– Porting to GPUs– Finding better solvers (what’s better often depends on the degree of
parallelization)
• Visualization• Workflows
8
Selected Highlight (1)• Large eddy simulations of extended wind farms, and direct numerical simulations
of turbulent channel – PI: Meneveau, Johns Hopkins– ECSS Project Team: Darren Adams and David Bock (NCSA)
• Interaction between large wind farms with multiple wind‐turbines and the atmospheric boundary layer flow
• The ECSS goals were to provide assistance implementing two dimensional domain decomposition and parallel FFT as well as 3D visualization representations to assist in simulations and analysis of large wind farms.
• The work plan was modified part way through to abandon an approach that included developing a stand‐alone MPI transpose and custom 2D FFT. Instead, the team modified the existing P3DFFT library to completely skip the FFT and transpose operations on the 3rd array index.
9
Performance of the 2D FFT in calculation the x and y derivatives. This was obtained from runs on the NICS Kraken system. The figure presents scaling of a 1,0243 mesh. The blue points are from the original 2D FFT implemented with 1D domain decomposition. The orange points are the new 2D MPI FFT scaled out to 8,192 processors.
Selected Highlight (2)• Visualization representations
– designing and developing a myriad of different techniques visualizing the changing vector field over time. Work focused specifically on various particle advection, tracing, and streamline
– development of high quality scenes integrating new particle visualization methods developed above with earlier existing volume rendering techniques to show multiple variables in a single representation.
10
Geosciences (Southern CA Earthquake Center)
• Spans gamut from largest simulations to midsize jobs to huge number of small jobs.
• Largest runs examine high frequency modes (short wave‐length, higher resolution) of particular interest to civil engineers, – need large distributed memory runs using
up to all cores of Ranger, Kraken.• Mid‐range core counts improve the velocity
model doing full 3‐D tomography– Clusters– Need large data available on disk (100 TB)
11
Excellent example of coordinated support‐
consultants at SDSC and PSC interface with those at
NICS, TACC, &NCSA to smooth migration of code. Improved performance 4x.
Goal - understand earthquakes and mitigate risks of loss of life and property damage.
SCEC-PSHA• Using the large scale simulation data,
estimate probabilistic seismic hazard (PSHA) curves for sites in southern California (probability that ground motion will exceed some threshold over a given time period).
• Used by hospitals, power plants, etc. as part of their risk assessment.
• For each location, need a Cybershake run followed by roughly 840,000 parallel short jobs (420,000 rupture forecasts, 420,000 extraction of peak ground motion‐ bundled into 40,000 actual jobs).
12
Managing these requires effective grid workflow tools for job submission, data management and error recovery, using Pegasus (ISI) and Dagman(U of Wisconsin Condor group).
FRED- Epidemiology Modeling Code
• Agent‐based modeling system that uses synthetic populations derived from census data to capture geographic and demographic distributions.
• Recently used in real time by the Dept. of Health and Human Services in managing the H1N1 outbreak
• Usually applied to a region. When scaled to the entire US, the serial implementation required approximately 96 hours and 540 GB of memory to complete a 100‐day simulation.
• An ECSS consultant reduced this to 3‐4 hours using less than 200 GB of memory.
13
15% serologic attack rate, no mitigation
Vaccinate a month before the peak
Support for Novel and Innovative Projects(led by Sergiu Sanielevici, Pittsburgh Supercomputing Center)
• Pro‐actively reaching out to communities new to advanced computing, e.g.– social science– social network analysis – language processing – genomics– digital humanities – economics – library science– public health
• Look for pioneers, work with them to craft the project, and then help them begin to execute it.
14
Large Genome Assemblies• ECSS Staff working with leading
researchers and code developers
• Largest ever metagenome assembly, using 3.5 TB RAM on PSC Blacklight
15
“I wouldn’t have been able to do anything on Blacklight without ECSS staff… (consultant) took a real interest and solved a lot of things that were hard for me. He found bugs in the software and got them resolved with the software authors. I’d worked for months and not made that progress. Without his expertise, I might have given up…”
The Never-Ending Language Learner (NELL)• Machine learning of new concepts from the
Web. • The project has collected 50M beliefs,
represented as triplets of the form <coffee, servedwith, donuts>.
• With ECSS, this corpus has been organized into a graph‐based knowledge storage system supported by the Neo4J graph database on the large shared memory system Blacklight.
• Can now efficiently explore the connectivity of this graph to infer new beliefs (Because Bob lives in Tokyo, Tom visits Japan)
Support for Community Codes(led by John Cazes, Texas Advanced Computing Center)
Community codes – applications, tools, and libraries used by multiple research groups• Deploying, hardening, and optimizing useful software systems
• Assisting users with community codes and tools• Establishing relationships between XSEDE and developer communities
• Extending XSEDE documentation to cover community code implementations
17
Collaboration with the Broad Institute: Genomics Community CapabilitiesPort and optimize Trinity and ALLPATHS‐LG for XSEDE resources • Initiated project supported 20 research groups• Adjusted workflow to work with parallel filesystems• Collaborated with the Broad Institute and the National Center for Genome Analysis to include parallel optimizations in the code base
• Success: Enabled the largest Trinity job ever– 20 assemblies with 1.5 billion reads each
18
Gadget- A Widely Used Astrophysics Code
• Sometimes when you change systems, and especially compilers, things that used to work, no longer do.
• Usually, this is because the author of the code had used what they thought was a default feature of a language, but it really wasn’t.
• In this case the consultant tracked down a serious error in Gadget code and eliminated an unsafe MPI call.
• The fixed version is now in the Gadget release, which is still in wide use.
19
Selected Highlight • Collaboration with the Broad Institute: Genomics Community
Capabilities• Internal project to provide support to over 15 research groups• Goal: Port and optimize the widely used genomics
applications, Trinity and ALLPATHS‐LG, to Blacklight and other XSEDE resources
• Challenges: Neither was designed for a parallel architecture• Solution:
– Adjusted the workflow to work with parallel filesystems– Collaborated with the Broad Institute and the National Center for Genome
Analysis to include parallel optimizations in the code base
• Success: Enabled the largest Trinity job ever– 20 assemblies with 1.5 billion reads each
20
Trinity Pipeline Inchworm64 cores ~100 hours
Chrysalis128 cores ~400 hours
Butterfly64 cores ~50 hours
Support for Science Gateways(led by Marlon Pierce, Indiana U)
• Science Gateways enable communities of users associated with a common discipline to use computational resources through a familiar and simpler interface e.g. a web interface.
• Examples of ECSS Science Gateway Support– Assist with job submission and data movement to XSEDE– Automation of scientific processes through workflows– Integration of XSEDE resources into a portal/gateway – Support with grid security and community accounts
22
Realizing the Universe for the Dark Energy Survey (DES)PI’s: August Evrard (UM) and Andrew Kravtsov (UC)
A synthetic 2x3 arcmin DES sky image showing galaxies, stars,and observational artifacts. Courtesy Huan Lin, FNAL.
The density of dark matter in a thin radial slice as seen by asynthetic observer located in the 8 billion light‐yearcomputational volume. Courtesy: Matthew Becker, Universityof Chicago.
• ECSS provided support to the Dark Energy Survey Simulation Working Group in developing large, multi‐staged computations (workflows) to develop Blind Cosmology Challenge (BCC) catalogs.
• These executions Catalog a variety of sky realizations in different cosmologies are analyzed, in a blind manner, by science teams generate expectations for galaxy yields in various cosmologies.
• Analysis of these simulated catalogs offers a quality assurance capability for cosmological and astrophysical analysis of upcoming DES telescope data.
Support for Training, Education, and Outreach(led by Jay Alameda, NCSA)
• Collaboration with XSEDE Training, Education, and Outreach Services
• Training support– On‐line course development and update– Synchronous training (live, web, …)– Collaboratively identify new training areas
• Education– Online course support
• Outreach– Conferences, speakers bureau, campus visits
24
Education: Applications of Parallel Computers• Adapted from Jim Demmel’s CS267 Course
– Video, Slides in Cornell’s Virtual Workshop– Quizzes in every class
• 3 Homework (programming) problems– ECSS staff helped port to XSEDE resources– ECSS staff part of support team for students– Autograding by Berkeley team
• Capped enrollment at 300 students– Limit reached in < 3 days
• Evolved into a Small Private Online Course (SPOC), in collaboration with a number of campuses.
25
XSEDE Technology Evaluation Database
• XTED:– publicly available– web‐accessible– database of technology projects
• www.xsede.org/xted• In the database:
– Registered Technologies• Technology creators can enter their own projects
– Evaluations of Selected Technologies• Contact : J. Ray Scott – [email protected]
26
September 4, 2015
XSEDE and Science Gateways
Using XSEDE Resources Through Your Web Browser
What Is a Science Gateway?
Image courtesy: http://goo.gl/Ftvxhc
Science Gateways
• Science gateways provide Web‐based access to XSEDE.– Help researchers and students run well‐known scientific application codes.
– Coming: provide access to online data sets
• Gateways are community provided resources• Gateways are domain specific
– Chemistry, phylogenetics, neuroscience, biophysics, …
What Are Some Example Science Gateways?
Some Prominent XSEDE Gateways
• CIPRES: phylogenetic research.– Prominent applications: RAxML, BEAST, MrBayes
• SEAGrid: computational chemistry, material science, and engineering– Prominent applications: Gaussian, LAMMPS, NWChem
• Neuroscience Gateway: computational neuroscience.– Prominent applications: NEURON, Freesurfer, Brian
• CyberGIS: geospatial research– Prominent applications: TauDEM, pPySAL, pRasterBlaster
Full listing: https://www.xsede.org/gateways‐listing
Do People Use Science Gateways?
More users access supercomputers via gateways than from the command line in 2014
2,398
2,984
0
500
1,000
1,500
2,000
2,500
3,000
3,500Active users Gateway users
NSF XSEDE users
Gateways Submit over 20,000 jobs/month to XSEDE supercomputers.
CIPRES: Supporting Research and Education• Most popular science gateway in
XSEDE– ~40% of all XSEDE users
• In use on 6 continents• Cited in major journals (Cell,
Nature, PNAS)• Used at major research institutions
(Stanford, Harvard, Yale)• Used by ~76 researchers for
curriculum delivery• Supports hundreds of publications
every year (1570 to date)• Used in 80% of EPSCoR states• Used by a 15‐year‐old high school
student who won the Massachusetts state science fair with no support from SDSC staff
How Do I Get Started? Can I Get Help? Can I Build My Own Gateway?
More Information
• Contact XSEDE Science Gateways Program Manager– Marlon Pierce, [email protected]– Marlon loves to get email about gateways.
• XSEDE Gateways Overview: https://www.xsede.org/gateways‐overview
• Community Mailing List: [email protected]– Send “subscribe gateways” to [email protected]
September 4, 2015
XSEDE Portal and Resources
XSEDE New User Training
Nowwhat?
Yeah! I got an Yeah! I got an XSEDE
allocation!
XSEDE User Portal (XUP)
• URL: portal.xsede.org• Single point‐of‐entry to information about XSEDE services and utilities for using them
• Anyone can create an XUP user account and access non‐project features
• Only XSEDE allocation project members can access project features
Using the XUP
• Create and login to your XUP Account• Use XSEDE resources responsibly• Get added to your XSEDE project• Navigate your personal My XSEDE webpage• Navigate the information in the XUP
Create and login to your XUP account
1. From the XUP homepage, click CREATE ACCOUNT
2. Complete the User Account Form
3. Verify your account request
4. Select your username and password
5. Login to the XUP
portal.xsede.org
Click the CREATE ACCOUNT link to access the XUP User Account Form
XSEDE Acceptable Use Policy
• Must accept the User Responsibilities Form after creating your XUP account and again at the beginning of each allocation you receive.
• Choose a strong password and protect it.• Close SSH terminals and log out of the User Portal when you
are finished with your session.• Report Suspicious Activity : email [email protected] or call 1‐
866‐907‐2383 immediately, regardless of the time of day.
XSEDE Cybersecurity Tutorialhttp://www.citutor.org
Get Added to Your XSEDE project
• PIs automatically have full access to their project’s account.
• The PI is responsible for managing users on their account.
• Ask the PI, or their allocation manager, to add your XUP username to the project.
Your My XSEDE webpage(1)WELCOME TO XUP• Quick access to
commonly used features.
(2)LATEST UPDATES• Latest information
specific to your user account.
(3)MY ACTIVE ALLOCATIONS• Summary of the active
projects for which you are either a PI or member.
Update your XUP User Profile
MY XSEDEProfile
View and or change your user information (organization, address). Make sure your email address is correct. XSEDE staff will use it to
communicate with you regarding your allocation.
Navigating the XUP
• My XSEDE• Resources• Documentation• Allocations
• Training• User Forums• Help• About
View the XSEDE Systems Monitor• Resources ‐> Systems Monitor Provides technical and status information for all of XSEDE's resources. The STATUS column indicates whether the system is up or down. If down, can click on status to find when the machine is expected to come back up.
Resources - Compute
49
Resources – Visualization, Storage, High Throughput Computing (HTC)
50
Resources - Software
51
Documentation – User Guides
52
Documentation – User Guide Sample
53
Allocations - Policies
54
Allocations: Request Submission
55
Allocations: Manage
56
Training
57
Accessing XSEDE ResourcesAuthentication Methods1. Password
• XUP credentials• Site-password• One-time password
2. Key-based
Single Sign-On• Enables logging in once to
access all of your allocated resources
Connection Methods1. GSI-OpenSSH2. OpenSSH
XSEDE SSO Login HubAn SSO enabled connection point to XSEDE resources
SSH to login.xsede.org using your XUP credentials
Move among resources using gsisshcommand
Managing your XSEDE files
1. Where to store files– Home directory– Scratch directory– Archival storage
2. How to move files– Command line using globus‐url‐copy, uberftp, scp, or sftp
– Globus Online
XSEDE File Systems• Home directory
– Location specified in the environment variable $HOME.– Use to store project files you want to keep long term such as source code,
scripts, and input data sets. – Not backed up regularly and not purged. – Quotas typically set to limit amount of disk space available.
• Scratch directory– Location specified in environment variable varies among resources but will
include the term SCRATCH, e.g. $SCRATCH_DIR.– Use to temporarily store files produced during application runs.– Not backed up and routinely purged.– No quotas. Available space depends on cumulative use by all users.
• Archival storage– Must request through allocation process
Moving Files - Globus Online• A fast, reliable, and secure file transfer service geared to the big data needs of the research community.
• Moves terabytes of data in thousands of files• Automatic fault recovery• Easy to use• No client software installation• Consolidated support and troubleshooting• Supports file transfer to any machine• Accounts are free ‐ www.globusonline.org
Globus Online Dashboard
Globus Online File Transfer
Choosing a file to move…
Running Jobs Overview
Compute Nodes
Home Scratch
File System
JobData
Data
Researcher
Login
Login Nodes
Command Line
Use for tasks such as fileediting, code compilation,
data backup, and job submission.
Read/write data from compute nodes to Scratch directory.
Store project filesSuch as sourcecode, scripts, andinput data sets toHome directory.
Run jobs by submitting your batch script to the compute nodes using the "qsub" command.
When you connect to a resource, you are on a login node shared by many users.
Your job is submitted to a queue and will wait in line until nodes are available. Queues are managed by a job scheduler that allows jobs to run efficiently.
BatchScript
Commands for code execution, copy input files to scratch,…Specify number/type of nodes,length of run, output directory, …
Login Nodes
• When you login to an XSEDE resource, you connect to a login node.
• Use login nodes for basic tasks such as file editing, code compilation, data backup, and job submission.
• Do not run compute jobs on the login nodes. This image cannot currently be displayed.
Where do I run compute jobs?
Running Compute Jobs
• Jobs are run on the compute nodes by submitting a batch script on a login node
• All jobs are placed in a batch queue after they are submitted.
• All XSEDE compute resources use a batch scheduler for running jobs.
• Resource User Guides on the XUP have details on your system’s scheduler.
Batch Scripts
• Batch scripts include scheduler specific directives, comments, and executable commands, e.g.:• Number and type of nodes needed• Time needed to run the job• Where to write output files
• Script commands are system specific – see the resource’s User Guide on the XUP for details
Batch Script for PSC’s Blacklight
1. #!/bin/csh2. #PBS -l ncpus=163. #ncpus must be a multiple of 164. #PBS -l walltime=5:005. #PBS -j oe6. #PBS -q batch7. set echo8. ja9. #move to my $SCRATCH directory10. cd $SCRATCH11. #copy executable to $SCRATCH12. cp $HOME/mympi13. #run my executable14. mpirun -np $PBS_NCPUS ./mympi15. ja -chlst
Blacklight uses the Portable Batch System (PBS) scheduler. Lines 2,4,5, and 6 are PBS directives.
Submitting Batch Scripts• Commands are machine specific, but follow general principles
• With PBS batch scripts, use the qsub commandqsub myscript.job
• Can also specify PBS directives as command‐line options:qsub -l ncpus=16 -l walltime=5:00 -j oe -q batch myscript.job
• Command‐line directives override directives in your scripts.
• portal.xsede.org Help Submit ticket
• portal.xsede.orgMy XSEDE Tickets– Submit ticket– View past tickets (both open and closed)
• Can also email [email protected] or call 1‐866‐907‐2383, at any hour (24/7)
Need help? Reporting and Tracking Issues
More “helpful” resourcesxsede.orgUser Services• Resources available at each Service Provider
• User Guides describing memory, number of CPUs, file systems, etc.
• Storage facilities• Software (Comprehensive Search)
• Training: portal.xsede.org Training• Course Calendar• On‐line training
• Get face‐to‐face help from XSEDE experts at your institution; contact your local Campus Champions.
• Extended Collaborative Support (formerly known as Advanced User Support (AUSS))
September 4, 2015
Thanks for listening and welcome toXSEDE!
