View
1
Download
0
Category
Preview:
Citation preview
Using Eclipse Parallel Tools Platform with BlueFern’s HPCs
The BlueFern Team
Module 1: Introduction
• Objective
• To introduce the Eclipse platform and PTP
• Contents
• What is Eclipse?
• What is PTP?
What is Eclipse?
• A vendor-neutral open-source workbench for multi-language development
• An extensible platform for tool integration
• Plug-in based framework to create, integrate and utilise software tools
Eclipse Features
• Full development lifecycle support
• Revision control integration (CVS, SVN, Git)
• Project dependency management
• Incremental building
• Content assistance
• Help
• Language sensitive searching
• Multi-language support
• Debugging
Parallel Tools Platform (PTP)
• The Parallel Tools Platform aims to provide a highly integrated environment specifically designed for parallel application development
• Features include:
• An integrated development environment (IDE) that supports a wide range of parallel architectures and runtime systems
• A scalable parallel debugger
• Parallel programming tools (MPI, OpenMP, UPC, etc…)
• Support for the integration of parallel tools
• An environment that simplifies the end-user interaction with parallel systems
• http://www.eclipse.org/ptp
Module 2: Installation
Objective
To learn how to install Eclipse and PTP
Contents
System Prerequisites
Eclipse Download and Installation of “Eclipse IDE for Parallel Application Developers”
Installation Confirmation
Updating the PTP within your Eclipse to the latest release
Before you continue, insert the disc we prepared for this workshop into your computer’s DVD
System Prerequisites
Local system (running Eclipse)
Linux (just about any distribution)
Mac OS X (10.5 Leopard or 10.6 Snow Leopard)
Windows (XP or above)
Java: Eclipse requires Sun or IBM Java
Only needs Java runtime environment (JRE)
Java 1.5 or higher
Java 1.5 is the same as JRE 5.0
Open JDK, distributed with some Linux distributions, should work successfully (it works for me)
The GNU Java Compiler (GJC), which comes standard on Linux, will NOT work
Eclipse Packages
The version of Eclipse for this workshop is 3.7, also known as “Indigo”
Eclipse is available in a number of different packages for different kinds of development
http://eclipse.org/downloads
With Indigo, there is a package directly relevant for HPC:
Eclipse IDE for Parallel Application Developers
This is recommended for all new installs
The latest packages, 3.7 SR2, are included on the disc you received earlier. They are within the eclipse_packages directory on the disc
PTP can also be added to an existing Eclipse installation
Eclipse Installation
• Download the “Eclipse IDE for Parallel Application Developers” package
• For this workshop, copy the file from the disc you received earlier (<DVD root>/eclipse_packages) and paste it in your home directory
• Make sure you match the architecture with the computer you are using
• For this workshop, please use “eclipse-parallel-indigo-SR2-incubation-macosx-cocoa-x86_64.tar.gz
• If your computer is running Linux or Mac OS X, untar the file
• On Mac OS X, you can just double-click in the Finder
• If your computer is running Windows, unzip the file
• Untarring or unzipping the package file creates an eclipse folder containing the executable as well as other support files and folders
Starting Eclipse
• Linux
• From a terminal window, enter:
“<eclipse_installation_path>/eclipse/eclipse&”
• Mac OS X
• From finder, open the eclipse folder where you installed it
• Double-click on the Eclipse application
• Windows
• Open the eclipse folder
• Double-click on the eclipse executable
Specifying A Workspace
• Eclipse prompts for a workspace location at the start up time
• The workspace contains all user-defined data
• Projects and resources such as folders and files
• Enter /Users/<logon id>/workspace for this workshop
Note: If you don’t want to see this message every time when Eclipse is launched, tick this check box
Eclipse Welcome Page
• The welcome page is displayed when Eclipse is run for this first time
• Select “Go to the workbench” to continue this tutorial
Check Installation Details
• To confirm you have installed Eclipse PTP successfully
• Mac: Eclipse>About Eclipse
• Others: Help>About
• Choose Installation Details
• Confirm you have “Eclipse IDE for Parallel Application Developers” in the installed software list
• Close the dialog: Close, OK
Checking for PTP Updates
• From time to time, there may be newer PTP releases than the Eclipse release
• PTP maintains its own update site with the most recent release
• Bug fix release can be more frequent than Eclipse’s and what is within the package
• You must enable the PTP-specific update site, so PTP updates can be found
Updating PTP (1)
• Enable PTP-specific update site
• Help>Install New Software…
• Click Available Software Sites link
• Ensure the checkbox is selected for the PTP site:
http://download.eclipse.org/tools/ptp/updates/indigo
• Choose OK
• Choose Cancel (to return to Eclipse workbench)
• Now select Help>Check for updates
• If you see “No updates were found”…
• It is only because there are no updates in the “Eclipse IDE for Parallel Application Developers”
• We will update the PTP within it
Updating PTP (2)
• We will get the latest PTP release (5.0.7), which is more recent than the version that is currently included (5.0.5) in the parallel package
• Update PTP via an archived package
• Download the package, ptp-master-5.0.7-201203231413.zip, from http://www.eclipse.org/ptp/downloads.php (included in the disc that you received earlier)
• Select Help>Install New Software…
• Select Add…
• Give this repository a sensible
name, such as PTP 5.0.7 update
• Select Archive…
• Navigate to and select the archived
package
• Click the OK button
Updating PTP (3)
• Alternatively, download the update from the Internet directly (skip this for this tutorial):
• Select Help>Install New Software…
• In the Work with: dropdown box, select the PTP update site that you confirmed already:
Updating PTP(4)
• Quick and dirty: (this is what we will do for this tutorial)
• Check everything – which updates existing features and adds a few more
• Detailed:
• Open each feature and check ones you want to update
• Icons indicator: Grey plug: already installed
• Double arrow: can be updated
• Colour plug: not installed yet
Updating PTP (5)
• When updating via a local archive,
unselect Contact all update sites during
install to find required software
• Select Next to continue updating PTP
• Select Next to confirm features to install
• Note: if you are NOT on Linux: you may have to go back and unselect, under Fortran Development Tools (Photran), the “Linux Intel Fortran Compiler Support” – only if it complains
• Accept the License agreement and select Finish
• Wait for installation to finish
• Select Restart Now when prompted
• We are Ready to go now!!!
Module 3: Developing with Eclipse
• Objectives:
• Learn basic Eclipse concepts: Perspectives, views, etc…
• Learn how to create and manage a C project
• Learn how to build and run a project
• Learn about navigating, searching, refactoring, etc…
• Contents:
• Basic Eclipse features
• Managing projects in Eclipse
• Creating a C project
• Build and run a C project
• Basic editor features
• Content Assist & Templates
• Navigating, Searching and Refactoring
Basic Eclipse Features
Eclipse Basics
• A workbench contains the menus, toolbars, editors and views that make up the main Eclipse window
• The workbench represents the desktop development environment
• Contains a set of tools for resource management
• Provides a common way of navigating through the resources
• Multiple workbenches can be opened at the same time
• Only one workbench can be open on a workspace at a time
view
view
view editor
Perspectives
• Perspectives define the layout of views and editors in the workbench
• They are task oriented, i.e. they contain specific views for doing certain tasks. For example:
• Parallel Runtime Perspective manages resource managers that are used for managing local or remote parallel runtime environments
• C/C++ Perspective for manipulating C/C++ source code
• Debug Perspective for debugging applications
• If you are still on the Welcome screen, select “Go to Workbench” now
Switching Perspectives
• Three ways of changing perspectives:
1. Choose the Window>Open Perspective menu option then select the target perspective or choose Other…
2. Click on the Open Perspective button in the upper right corner of the screen (hover over it to see names)
3. Click on a perspective shortcut button
• Try to switch to the C/C++ Perspective now.
Which Perspective?
• Which perspective am I in?
It is indicated on the Title Bar
Views
• The workbench window is divided up into Views
• The main purposes of a view are:
• To provide alternative ways of presenting information
• For navigation
• For editing and modifying information
• Views can have their own menus and toolbars
• Items available in menus and toolbars are available only in that view
• Menu actions only apply to the view
• Views can be resized
Stacked Views
• Stacked views appear as tabs
• Selecting a tab brings that view to the foreground
Expand a View
• Double-click on a view/editor’s tab to fill the workbench with its content
• Repeat to return to original size
Double Click
Double Click
Help
• To access help:
• Help>Help Contents
• Help>Search
• Help>Dynamic Help
• Help Contents provides detailed help on different Eclipse features in a browser
• Search allows you to search for help locally, or using Google or the Eclipse web site
• Dynamic Help shows help related to the current context (perspective, view, etc.)
Eclipse Preferences (1)
• Eclipse Preferences allow customization of almost everything
• To open use
• Mac: Eclipse>Preferences…
• Others: Window>Preferences…
• The C/C++ preferences allow many options to be altered
• In this example you can adjust what happens in the editor as you type.
Eclipse Preference (2)
• In this example, the Code Style preferences are shown
• Changing preferences here will allow code to be automatically formatted in different ways
Managing Projects in Eclipse
Project Types
• There are three types of project in Eclipse:
• Local
• Source is located on local machine, builds happen locally
• Synchronized
• Source is local, then synchronized with remote machine(s)
• Building and launching happens remotely and can also happen locally
• Remote
• Source is located on remote machine(s), build and launch takes place on remote machine(s)
• We will concentrate on the Local project for the rest of Module 3 and revisit Synchronized and Remote in later modules.
Project Build Types
• A project’s build type can either be:
• Makefile-based: Project contains its own makefile (or makefiles) for building the application
• Managed: Eclipse manages the build process, no makefile is required
• Time to start our Hello World project !!!
Create a C Project
Creating a Project (1)
• Select File>New>Project…
• Select C Project then press Next>
• You can combine the these two
steps by selecting the shortcut
File>New>C Project
Creating a Project (2)
• Select Hello World ANSI C Project under the Executable project type
• Select a Toolchain based
on what’s available on your local computer. (select MacOSX GCC for this workshop because we are using Macs)
• Give the project a name, such as HelloWorld
• Press Next> if you wish to configure basic project settings; otherwise press Finish (press Next> for this workshop)
Creating a Project (3)
• Update basic settings as you wish
• Press Next> when you are ready to update the project’s configurations
• Or press Finish if you wish to skip the project configuration
• (Press Next> for this workshop)
Creating a Project (4)
• Tick on Configurations that you wish to add to this project. (Tick both Debug and Release for this workshop)
• Select Finish to complete the creation of a project
• Selecting Advanced settings… will allow you to configure more project settings. We will skip this for this workshop.
Creating a Project (5)
A Hello World Project in a Workbench
Build and Run a C Project
Build a Project
• To build the current project using the current active build configuration, click on icon
• You can build using a different build configuration by clicking the down-arrow on the icon and selecting the configuration you want to use
(The current active build configuration is ticked)
• The build output is displayed in the Console tab view
Run a Project
• Click icon to run the current project
• If the project has not been built or the current build is not up to date, it will be automatically built using the current active build configuration
• When running the project, all data printed to the standard output will be displayed in the Console tab view (Except if you are using the Cygwin toolchain under a Windows OS, in such a case, you will have to run the produced executable manually within a Cygwin Terminal)
Basic Editor Features
Editor
• An editor for a resource (e.g. a file) opens when you double-click on a resource
• The type of editor depends on the
type of the resource
• .c files are opened with the
C/C++ editor by default
• You can use Open With to use
another editor
• In this case the default editor is fine (double-click)
• Some editors can edit more than just raw text
• When an editor opens on a resource, it stays open across different perspectives
• An active editor contains menus and toolbars specific to that editor
Saving File(s) in an Editor
• When you change a file in the editor, an asterisk (*) on the editor’s title bar indicates unsaved changes
• Save the changes by using
shortcut Command/Ctrl+S
or File>Save or icon on
the tool bar
• Save multiple files by using
shortcut Command/Ctrl+Shift+S
or File>Save All or icon on the tool bar
Editor and Outline View
• Double click on the source file, its default editor will open on the main view
• Outline view is shown for the file in the editor
• Console tab view shows the result of a local build, run etc…
Source Code Editor and Markers (1)
• Modify HelloWorld.c so the
content is the same as shown in
the screenshot
• Save these modifications
(hint: Command/Ctrl+S)
• Click icon on the tool bar to
build the project or
select Project>Build Project
(Yes, there is an error in these modifications and it is intentional)
Source Code Editor and Markers (2)
• A source code editor is a special
type of editor for manipulating
source code
• Language features are
highlighted
• Marker bar for showing
• Breakpoints
• Errors/warnings
• Task Tags, Bookmarks
• Location bar for navigating to interesting
features in the file Icons:
Source Code Editor and Markers (3)
• Resolve the error by adding a semicolon (;) to the end of puts function
• Rebuild the project
• All error and warning markers should be disappear
• When you see bug icons in the editor marker
bar, they are likely suggestions from Codan
• Code checkers can flag possible errors,
even if code is technically correct
• To turn them off, use Preferences
Window>Preferences or
Mac: Eclipse>Preferences
C/C++ > Code Analysis and
uncheck all problems
• Select Ok to close preferences
Source Code Editor and Markers (4)
• Text editors can show line
numbers in the left column
• To turn on line numbering:
• Right-mouse click in the
editor marker bar
• Click on
Show Line Numbers
Adding Existing Files to a Project (1)
• Right click or (Mac: Ctrl+click)
on the src folder in
the Project Explorer and then
select Import…
Or
• Select the src folder in
the Project Explorer and then
select File>Import…
Adding Existing Files to a Project (2)
• Type “file system” in the
Select an import source: text field
• Select , then
click Next> button
• Click Browse… button and navigate to
<DVD root>/tutorial/Module_03
• Tick both Func.c and Func.h
• Click Finish button
Content Assist & Template
Content Assist & Template in Source Editor (1)
• Use Content Assist to include Func.h in HelloWorld.c:
• In the source editor, type
#include “
• Press Ctrl+Space to enable Content Assist
• Available headers should be displayed in a list
• Type Fu, suggestion list shows only “Func.h”
• Press Enter to complete the include statement
Content Assist & Template in Source Editor (2)
• Use Content Assist to call Func(int a) in HelloWorld.c’s main function:
• Press Ctrl+Space to enable Content Assist
• A long suggestion list is shown
• Keep typing Fun, and you will see the suggestion list gets shorter
• Use arrow keys to select the function you wish to call. In this case, select Func(int a):int
• Press Enter to accept the selected function
Content Assist & Template in Source Editor (3)
• Content Assist also helps you to peek at the signature of a function
• As you select Func(int a):int, its signature will be automatically displayed
• If you didn’t use Content Assist to write a function, you can still use it to check its signature by pressing Ctrl+Space after typing the function name
Content Assist & Template in Source Editor (4)
• Use Content Assist &
Template to write a
For loop:
• Type: for then press
Ctrl+Space
• Select for – for loop then press Enter
• A for loop template will be written in the source code
• Enter i to replace the var in the template
and then press Tab
• Enter 10 to replace the max in the template and press Tab to complete the template
Content Assist & Template in Source Editor (5)
• Use Content Assist & Template to modify your HelloWorld.c to match what is shown in the screenshot below:
Navigating, Searching and Refactoring
Source Code Navigation (1)
• Eclipse’s source code editor provides some handy features to navigate through your code
• On demand hyperlink:
• On line 23 in HelloWorld.c, a function Func that is not implemented within the current file is called
• To open the file where Func is implemented and navigate to it
• Hold the Ctrl/Command key and click on the Func
• On demand hyperlink also works on functions and variables in the current file
• Hold the Ctrl/Command key and click on variable msg in line 23, you will see the cursor is now moved to line 18 where msg is declared
(Note: Click = left mouse click on Windows/Linux)
Source Code Navigation (2)
• Open declaration
• Move the cursor to the middle or after a variable or function name
• Right-Click or (Ctrl+Click) to bring up the context menu then select Open Declaration
• or you can just press the shortcut F3
F3
Simple Find and Replace
• Ctrl+F or (Command+F) to bring up the simple
Find/Replace dialog
• It only works within the current active file in the
editor view
• Try to find all references of Func in HelloWorld.c
Language Specific Searching (1)
• Select Search>C/C++… or Ctrl/Command+H and click the “C/C++Search” tab
• It “knows” what can be
declared in C/C++
• Try to search for all
occurrences of function
Func in the current
Workspace
Language Specific Searching (2)
• The outcome of the search will be displayed in the Search tab view
• Expand the tree to see all occurrences
• Double click a matched occurrence to navigate to its location in a source file
Refactoring (1)
• Eclipse features a Refactor tool which helps you to modify your code and preserve its correctness at the same time:
• It only works on specific types of modifications, such as renaming a function, extracting selected lines of code into a function etc…
• Bring up the Refactor menu by selecting
Refactor in the menu bar or
Right Click or Ctrl+Click on the editor to open
the context menu and then select the Refactor
menu
Refactoring (2)
• Hard coded constant within a
function is bad in practice. Let’s
extract it to a constant
• Select the number 10 in line 22 and then bring up the Refactor menu
• Select Extract Constant…
• Give the new constant a meaningful name
then click the Next> button to preview
the outcome
Refactoring (3)
• Check the code comparisons and verify if the Refactor is doing what you are expecting
• Press the Finish button to confirm the change
Refactoring (4)
• The final code should be like the following:
Module 4: Working with MPI C Code in Eclipse
• Objectives
• Learn how to create a MPI project using C
• Learn about MPI features
• Learn how to build an MPI program using C on a local system
• Contents:
• Configure MPI Settings in Eclipse
• Helpful features for using MPI
• Build and Run a MPI Job Locally
• Build and Run a MPI Job Remotely
Configuring MPI Settings in Eclipse (1)
• You need to configure Eclipse to ‘find’ your local MPI install to work with a MPI based project properly. If it is not configured, you won’t be able to build and run your code locally.
• Select Window>Preference or (Eclipse>Preference on MacOS) to bring up the preference dialog
• Expand Parallel Tools, Parallel Language Development Tools and then select MPI
Note:
1. It is assumed that a version of MPI is already installed in your local environment
2. If you are using Eclipse and the Cygwin toolchain in a MS Windows environment, skip this section. BlueFern is not currently supporting MPI in a MS environment at this stage due to technical challenges
Configuring MPI Settings in Eclipse (2)
• Press the New… button to bring up a file chooser and navigate to the directory where MPI’s header is stored
(For this tutorial or on a Mac, navigate to /usr/include)
• Set MPI build Command(C): to mpicc
• Set MPI build Command (C++): to mpiCC
• Press the OK button to close the preference dialog
Creating a MPI Hello World Project (1)
• Select File>New>C Project
• Enter HelloMPI in the Project name
field
• Select MPI Hello World C Project
as the Project Type
• Select the appropriate toolchain that is
available for the local development
environment
(MaxOSX GCC for this workshop)
• Press the Finish button to create the
project
Creating a MPI Hello World Project (2)
• Double click HelloMPI.c in the project’s src folder to open it in the source editor view
Helpful Features for Using MPI
Context Sensitive Help (1)
• Eclipse provides a context sensitive help, and it works for MPI functions too
• Select Help>Dynamic Help
• Click on the Restore shortcut to prevent it from
minimising the view
• Click on line 28, MPI_Comm_Rank...
• Special info for this function is now
displayed in the help view
• Click on the link to open the function’s
help document Click here
Context Sensitive Help (2)
• Hovering the mouse pointer over a MPI function brings up a short description for the function
Parallel Analysis
• Eclipse PTP can parse code in the current selected files or project to provide parallel analysis
• One analysis identifies all MPI artifacts and displays them in a tab view
• Select the HelloMPI project
• Click on the downward arrow on icon and select
Show MPI Artifacts
• The outcome of the analysis is displayed in the
MPI Artifact View, clicking on the artifact will take you the
line where the artifact resides in the source code
Building and Running a MPI Project Locally
(Note: Skip this section if you are using PC running MS Windows OS
and do not have Open MPI installed)
Build a MPI Project Locally
• If the Eclipse has been configured correctly, you should be able to build the project successfully by clicking the icon on the toolbar
• The build output is displayed in the Console tab view
• If the project does not build successfully, check if the project’s properties are set correctly
Checking Project Properties
• Select Project>Properties
• Expand C/C++ build,
• Select Tool Chain Editor, For this workshop
• the Current toolchain should be MacOSX GCC
• the Current builder should be Gnu Make Builder
• Select Settings, For this workshop
• the Command for the MacOS X C Linker should be mpicc
• the Command for the GCC Assembler should be as
• the Command for the GCC C Compiler should be mpicc
(Note: Project properties are build configuration specific, please make sure you check all of them in all available build configurations)
Resource Manager to Control a Local Open MPI Runtime (1)
• Switch to Parallel Runtime perspective
• Select Windows>Open Perspective>Others...
• Select Parallel Runtime, and click the OK button
Resource Manager to Control a Local Open MPI Runtime (2)
• Right Click or (Ctrl+Click on Mac)
within the Resource Managers tab
view to bring up its context menu
and select Add Resource Manager...
• Select Open MPI and then click on
the Next> button
Resource Manager to Control a Local Open MPI Runtime (3)
• Select Local, for both
Remote service provider: and
Connection name:
• Click on the Finish button to create
a local Open MPI resource manager
using default settings
(Note: default Open MPI settings are
sufficient for this tutorial)
Resource Manager to Control a Local Open MPI Runtime (4)
• Select the newly created
resource manager,
Open_MPI@Local (Open MPI)
• Right Click or
(Ctrl+Click on a Mac)
• Select Start Resource Manager to start the resource manager
• If the resource manager is started successfully, its icon should turn to green
• Switch back to C/C++ perspective by using its
shortcut on the toolbar
Run Configuration for a Local MPI Job (1)
• To create a run configuration using the local Open MPI runtime
• Click on the down arrow next to the Run As... icon on the
toolbar
• Select Run Configurations...
• Select Parallel Application as the
configuration type
• Click on the to add a new configuration
Run Configuration for a Local MPI Job (2)
• Give the configuration a
name (use HelloMPI_Local
for this tutorial)
• Select Open_MPI@Local
as the Resource Manager:
• Set Number of processes:
to 4
• Click on the Apply button
to save the configuration
Run Configuration for a Local MPI Job (3)
• Click on the Application tab
• Click on the Browse button
next to the
Application Program: text
field
• Navigate to the directory
where the built executable is,
select the executable and
click on the OK button
(Note: for this tutorial, select
HelloMPI in <Eclipse Workspace Dir>/HelloMPI/Debug/)
Running a MPI Job Locally (1)
• After you complete the run
configuration, the Run button
should be enabled
• You can click on the Run
button to launch a run using
the current configuration
• For this tutorial, click the
Close button and we will
launch the run from the
workbench instead
Running a MPI Job Locally (2)
• To launch a run using a specific Run Configuration from the workbench
• Click on the down arrow next to the Run As icon on the toolbar
• Select the Run Configuration you wish to launch the run
(For this tutorial, select HelloMPI_Local)
• The standard output generated by the run
will be displayed in the Console tab view on
the workbench
Building a MPI Project on BlueFern’s HPCs
Converting an Existing Project to a Synchronized Project (1)
• Switch to the C/C++ Perspective if your workbench is in another perspective
• Select File>New>Other...
• Type Convert in the Wizard:’s
text filter
• Select
Convert C/C++ or Fortran Project
to a Synchronised Project
• Click on the Next> button
Converting an Existing Project to a Synchronized Project (2)
• Select HelloMPI as a candidate
for conversion
• Select Git as the
Synchronization Provider
• Select Remote Tools as the
Remote Provider:
• Click the New... button to
create a connection to the remote
environment
Converting an Existing Project to a Synchronized Project (3)
• Give the new connection a Target name:
• Tick the Remote host radio button
• Enter the IP address or host name of the
remote host in Host:
(For this workshop, enter
hpclogin1.canterbury.ac.nz)
• Enter your user name and password
for logging on to the remote host
(they are supplied to you in the beginning
of this workshop)
• Click on the Finish button to create the
connection
Converting an Existing Project to a Synchronized Project (4)
• Enter the location where you wish
to store the current project on
the remote host
(you can select the Browse... Button
to open a file browser viewing the
remote file system)
• Click on the Finish button to convert
the current project to a
synchronised project
• A new build configuration should now
be available to this project
Configuring Remote Building Environment (1)
• If you try to build the project using the Remote build configuration, Eclipse will automatically upload your local files to the remote host and then invoke a build
• However, the build will most likely fail. This is because the build environment on the remote host is different from your local environment
• For this tutorial, you need to update:
• Toolchain,
• Settings for compiling and linking
• Environment variables
• Building tool
Configuring Remote Building Environment (2)
• Right click or (Ctrl+Click) on the HelloMPI project in the Project Explorer tab view to open its context menu
• Select Properties in the context menu
• Expand C/C++ Build properties
• Select the Tool Chain Editor
Configuring Remote Building Environment (3)
• Project properties is build configuration specific, change the current configuration to Remote, if it is not currently active
• Unselect Display compatible toolchains only checkbox
• Change Current toolchain: to XL C/C++
• Change Current builder: to
Sync Builder
• Click on the Apply button to
persist the change
Note: BlueFern encourage users
to use XL C/C++ compilers when
using BlueFern’s HPCs.
Configuring Remote Building Environment (4)
• Select Synchronize properties
• Confirm its properties are using the remote
connection created earlier
• If you wish to update the current project to
synchronise and utilise a different remote
environment, you will need to update these settings
(skip this for this tutorial)
• Select the Settings to setup compiler and linker
Configuring Remote Building Environment (5)
• Select Local XL C Compiler, and then enter mpcc in the Command:
• Select Local XL C
Executable Linker,
and then enter mpcc in
the Command:
• There are other options for
configuring the compiler
too, but we will skip them
for this tutorial
• Click on the Apply button
(Note: mpcc is a compiler driver to compile and link MPI C code. Using it saves us from entering paths to MPI headers and libraries)
Configuring Remote Building Environment (6)
• Select the Environment properties. This is where you can setup environment variables in the Remote Environment in build time
• They default to your local environment settings and are most likely
not compatible in the
remote environment
• For this workshop,
Undefine all
variables. We will
use SSH session’s
default instead
• Click the Apply
button when done
Configuring Remote Building Environment (7)
• Finally select the C/C++ Build properties
• Unselect the check box next to Use default command
• Change the
Build command:
to gmake
• Click on the OK
button
Note: Eclipse generates makefiles in GNU format when a project is set as a Managed project. Therefore you need to use GNU make to build the project, which is gmake under AIX
Build a Project in a Remote Environment
• Click on the down arrow on the build icon on the toolbar and select 3 Remote (Build on remote machine)
• It triggers Eclipse to synchronize the local
source code with the configured remote host
and also invoke the build
• The outcome of the build is displayed in the Console tab view on the workbench
Launch a MPI Job on BlueFern’s HPCs
Resource Manager to Submit a Job via LoadLeveler (1)
• Switch the workbench to the Parallel Runtime perspective
using a shortcut on the toolbar or
select Window>Open Perspective>Other...
and select Parallel Runtime
• Click within the Resource Managers view,
Right Click or (Ctrl+Click on a Mac),
and select Add Resource Manager...
Resource Manager to Submit a Job via LoadLeveler (2)
• Select IBM LoadLeveler as
the Resource Manager Type:
• Click on the Next> button
(Note: Although it is possible to use
other types of resource manager with
BlueFern’s HPCs, IBM LoadLeveler is
the recommended resource manager)
Resource Manager to Submit a Job via LoadLeveler (3)
• Select Remote Tools as the
Remote service provider:
• Select the remote connection created earlier for Connection name:
• Enter
/usr/local/bin/ptp_ibmll_proxy
in Path to proxy executable:
• Click on the Finish button
Note: we will be using default configurations for the LoadLeveler proxy. They can be changed by clicking on the Next> button and following the rest of the wizard
Resource Manager to Submit a Job via LoadLeveler (4)
• Right Click (or Ctrl+Click on Mac) the newly created resource
manager in the Resource Managers and select
Start Resource Manager
• Status of the remote cluster
is now displayed in the
Machines tab view
Prepare a LoadLeveler Script (1)
• Switch the workbench back to C/C++ perspective
(hint: shortcut or Windows>Open Perspective>(Others...))
• Right Click (or Ctrl+Click on Mac) on the src folder under HelloMPI project
• Select New>File
Prepare a LoadLeveler Script (2)
• Enter HelloMPI.ll in the File name: field
• Click on the Finish button
Prepare a LoadLeveler Script (3)
• Prepare your LoadLeveler script as shown below:
Note: if you are not sure about what you value need to populate for the initialdir, you can get it from the Root Location: under the synchronize setting in project
properties
Run Configuration for a Remote MPI Job (1)
• To create a run using the local Open MPI runtime
• Click on the down arrow next to the Run As... icon on the
toolbar
• Select Run Configurations...
• Select Parallel Application as the
configuration type
• Click on the to add a new configuration
Run Configuration for a Remote MPI Job (2)
• Give the configuration a
name
(use HelloMPI_Remote
for this tutorial)
• Select
IBM@BlueFern.....
as the Resource Manager:
• Click on the Browse
button to open a file
browser on the remote file
system, and navigate to
the LoadLeveler script prepared earlier
Run Configuration for a Remote MPI Job (3)
• Click on the Application tab
• Click on the Browse button
next to the
Application Program: text
field
• Navigate to the remote
directory where the built
executable is,
select the executable and
click on the OK button
• Click the Apply button to
save the current configuration
Running a MPI Job Remotely (1)
• After you complete the run
configuration, the Run button
should be enabled
• You can click on the Run
button to launch a run using
the current configuration
• For this tutorial, click the
Close button and we will
launch the run from the
workbench instead
Running a MPI Job Remotely (2)
• To launch a run using a specific Run Configuration from the workbench
• Click on the down arrow next to the Run As icon on the toolbar
• Select the Run Configuration you wish to launch the run
(For this tutorial, select HelloMPI_Remote)
• Unlike using Open MPI to run a local MPI job,
your job will be processed by the LoadLeveler in
batch mode.
• When the job’s run is completed, its output will be stored in the file specified in the LoadLeveler script
• If you wish to be notified when your job’s run is completed, populate your email address in the #@ notification field in your LoadLeveler script
Running a MPI Job Remotely (3)
• Because the project has been converted to a Synchronized project, files generated in the remote file system will be automatically copied to your local file system
• When the job’s run is completed, you should be able see its output files displayed in the Project Explorer view on the workbench if these files are generated within the project’s root directory or sub directories
• Double click such files to check the run’s output
Module 5: Working with a Real Software Project
• Objectives
• Learn how to use Eclipse PTP for a real software project
• Contents:
• Create an empty project
• Import existing code into a project
• Set up MPI runs locally and remotely
Create an Empty Synchronized Project (1)
• Create a Synchronized project because the software will be run on a remote HPC mostly
• File>New>Project...
• Select Synchronized C/C++ Project under the Remote category
(hint: use the text filter to find it quickly)
• Give the project a project name
• Select MPI Empty C Project as the project type
• Select a Linux GCC as the project’s tool chain
Note: Although we actually want to use the XLC C/C++ Tool Chain in the remote environment, there are some issues when using it in the project creation wizard. The workaround is to select Linux GCC in project creation and change it later
Create an Empty Synchronized Project (2)
• Click on the Next> button to skip the MPI Project Settings and Build configuration dialogs
• Select Remote Tools as the Remote Provider
• Select the remote connection created previously as the connection
• Enter the location where you want to store the project on the remote host
(Note: you may want to include the project name in the path here. Eclipse does not crate a new project in the remote location for you)
• Click on the Finish button to complete the creation of an empty project
Create an Empty Synchronized Project (3)
• The newly created project will have three build configurations:
• Debug – synchronise local files with the remote files and then invoke a debug build in the remote environment
• Release – synchronise local files with the remote files and then invoke a release build in the remote environment
• Workspace– synchronise local files with the remote files and then invoke a debug build in the local environment
Importing Existing Source Code to the Project
• Select the newly created project in the Project Explorer, bring up its context menu and create a new folder called src for the project
(Note: It is a good practice to keep all source code files in a folder under the project’s root directory. It separates source code files from other types of files, such as documentation.)
• Use the import utility to import all files in
/<DVD Dir>/tutorials/Module_5/ to the project’s src folder
• Select the src folder in the Project Explorer
• Bring up the context menu (right click or ctrl+click on a Mac)
• Select Import...
• Follow the wizard to complete this task
Updating Project Properties (1)
• Select the project in the Project Explorer
• Bring up its context menu and select Project Properties
• Project properties are build configuration specific, we need to update the project properties in all configurations
• Update the project properties based on the tables below:
• C/C++ Build>Tool Chain Editor:
Build Configuration
Debug Release Workspace
Toolchain XL C/C++ Tool Chain
XL C/C++ Tool Chain
Linux GCC (MacOSX GCC)
Builder Sync Builder Sync Builder Sync Builder
Updating Project Properties (2)
• C/C++ Build>Settings:
Build Configuration:
Debug Release Workspace
Local XL C++ Compiler
mpCC mpCC GCC C++ Compiler
mpiCC
XL C++ Executable Linker
mpCC mpCC GCC C Compiler
mpicc
GCC C++ Linker
mpiCC
Updating Project Properties (3)
• C/C++ Build>Environment
• Undefine all environment variables for both Debug and Release build configurations
• C/C++ Build
• Change the Build command: to gmake, because Eclipse generates makefiles in GNU format for managed projects
Building the Project
• Try building the project remotely using both Debug and Release configurations and both of them should result in a successful build
• Try building the project locally using the Workspace configuration and it will fail, because the software uses a library called mass, which is not available on your local computer
Fixing The Local Build (1)
• mass (Mathematical Acceleration Subsystem) is an IBM tuned version of the C math library that provides better performance on IBM HPCs
• To run this project locally, we need to use GNU C’s standard math library instead
• Let’s try to use a macro and change the Workspace build configuration to control the build behaviour
• Double click on the build error displayed in the Console tab view on the workbench to open the file and the location where this issue resides
Fixing The Local Build (2)
• Modify the beginning of fie agonist_variation_main.cpp as in the following:
(This simple macro basically says: if symbol MATH_H is defined when compiling the source code, include math.h; otherwise include mass.h instead)
Fixing The Local Build (3)
• Use the searching tool to find out if mass.h is defined in any other files
• Select Search>File... to search multiple files
• Entering mass.h in the Containing text: and then click on the Search button
• The search outcome will be displayed in the Search tab view on the workbench
• You will find that mass.h is also included in
update.cpp
• Double click update.cpp in the search tab
view to open the file in the editor
Fixing The Local Build (4)
• Use the same macro to control the build behaviour when file update.cpp is complied
• Introduce the following changes to file update.cpp:
Fixing The Local Build (5)
• Define symbol MATH_H in the build configuration Workspace
• Open the project’s properties
• Change the build configuration to Workspace
• Select C/C++ Build>Settings
• Select Preprocessor under GCC C++ Compiler
• Click on the icon in the panel of Defined Symbols
• Enter USE_MATH_H and then click on the OK button
• Click the Apply button to save the changes
• Click the OK button to close the project properties dialog
• Try building this project using the Workspace configuration again, the project should build successfully now.
Creating a Local Run (1)
• The sample code used in this tutorial hard coded the size of its simulating model by a #define symbol, let’s find it and replace it with a smaller value (otherwise, you will be waiting for quite some time for the run to complete)
• Use the searching utility to find symbol NUM_EC
• Set its defined value to 4
• Save the change
• Rebuild the project
Creating a Local Run (2)
• When running this sample code, it will generate a lot of output files. It would be an unnecessary overhead to synchronise such files with the remote file system. Let’s exclude them from been synchronised:
• Identify this project in the Project Explorer
• Expand its project tree
• Select the Workspace folder
• Right click or (Ctrl+Click on a Mac) on the Workspace folder to bring up its context menu
• Select Synchronization>Exclude Selection
Creating a Local Run (3)
• Open the Run Configuration wizard
• Create a new configuration for a
Parallel Application
• Under the Resources tab
• Select a local Resource Manager for this configuration (Note: use Open_MPI@Local for this workshop)
• Set the number of processes available to this run to 4 (Number of processes field when using a local OpenMPI resource manger)
• Under the Application tab
• Check if the Parallel Project field has been populated correctly, if not correct it
• Enter the full path to the executable built locally in the Application program field
Running the Project Locally
• After setting up a run configuration, click on the Run button to launch it
• the button might not be enabled if the selected Resource Manager is not currently running
• In such a case, save the current configuration and close the wizard
• Switch the workbench to the Parallel Runtime perspective and start the resource manager selected for the run configuration
• Go back to the run configuration and launch the run
• When the run is completed, you will see END OF PROGRAM in last line of the Console tab view
• The files generated by the run will be available in the Workspace folder under the project in the Project Explorer view on the workbench
Creating a Remote Run (1)
• Try to build the project using both the Debug and Release build configurations. The project should build successful in both configurations
• To avoid unnecessary synchronisation overhead, exclude both Debug and Release folders under the project
• For this tutorial, a remote run will be managed by the LoadLeveler. So, we need to prepare a LoadLeveler script first
• Add a new file to the project and call it remote_run.ll
• Open the newly created file in the source code editor by double clicking it in the Project Explorer view
• Create the script based on the following:
Creating a Remote Run (2)
# Example MPI LoadLeveler Job file # @ shell = /bin/ksh # # @ job_name = Agonist_Variation # # @ job_type = parallel # # # @ node = 1 # @ tasks_per_node = 4 # # # @ wall_clock_limit = 00:00:05 # # @ group = tutorial # # set initial dir to the path to your project in the remote environment + /Release/ # initialdir = /hpc/home/<path to the project in the remote environment>/Release # @ initialdir = /hpc/home/ysu20/workspace/Agonist_Variation/Release # # @ output = $(job_name).$(jobid).out # @ error = $(job_name).$(jobid).err # @ notification = never # @ class = dev8_1 # @ queue # invoke the executable poe ./Agonist_Variation
Creating a Remote Run (3)
• Creating a remote run configuration:
• Open the Run Configuration wizard
• Create a new configuration for a
Parallel Application
• Under the Resources tab
• Select a remote Resource Manager for this configuration (Note: use IBMLL@A_Remote_Connection for this workshop)
• Use Advanced Submit mode
• Enter the path and filename of the Loadleveler script that was created earlier
Creating a Remote Run (4)
• Under the Application tab
• Check if the Parallel Project field has been populated correctly, if not correct it
• Enter the full path to the executable built locally in the Application program field
• After the configuration is created, launch the run
• The job will be processed by the LoadLeveler in the remote environment in batch mode
• The run’s standard output will not be displayed in the Console tab view
• The best way for getting a notification is to populate your email address in the notification field in the LoadLeveler script, particularly if the run can last for a long time
(For this tutorial, the run will only take a few minutes. Let’s just wait for a message saying the remote job is terminated in the Console tab view)
Obtaining Output from a Remote Run
• A practical parallel run usually generates a large amount of data, stored in the remote file system as one or more files, therefore it is not practical to use a project’s file synchronisation feature to retrieve such files
• Eclipse uses git for file synchronisation and it is not designed for transferring files large in size
• BlueFern recommends its users to use:
• FileZilla, a tool with an easy to use GUI and supports SFTP
• scp command that is available on most Linux/Unix/MacOS environments (this is available on a Windows environment if you have Cygwin installed)
Getting More Help for Using Eclipse PTP
• Eclipse PTP User Group:
• Go to https://dev.eclipse.org/mailman/listinfo/ptp-user and subscribe to the mail list
• Send your questions to ptp-user@eclipse.org
• Contact BlueFern via bluefern@canterbury.ac.nz
• We will do our best to answer questions related to how to use Eclipse PTP with BlueFern’s HPCs
Acknowledgements
• This tutorial was developed based on the tutorial, “A New and Improved Eclipse Parallel Tools Platform: Advancing the Development of Scientific Applications”, which was presented by Greg Watson (IBM), Beth Tibbitts (IBM), Jay Alameda (NCSA), Jeff Overbey (UIUC), Wyatt Spear (U. Oregon) and Alan Humphrey (SCI) in SC11
• The original tutorial can be downloaded at http://download.eclipse.org/tools/ptp/docs/ptp-sc11-slides-final.pdf
• The sample code used in Module 5 is provided by Dr. Mohsin Shaikh and the Brains Trust research group in the University of Canterbury
The END and Thank You
Recommended