Research Article Using Coarrays to Parallelize Legacy Fortran …downloads.hindawi.com/journals/sp/2015/904983.pdf · 2019. 7. 31. · Research Article Using Coarrays to Parallelize

Research ArticleUsing Coarrays to Parallelize Legacy Fortran ApplicationsStrategy and Case Study

Hari Radhakrishnan1 Damian W I Rouson2 Karla Morris3

Sameer Shende4 and Stavros C Kassinos5

1EXA High Performance Computing 1087 Nicosia Cyprus2Stanford University Stanford CA 94305 USA3Sandia National Laboratories Livermore CA 94550 USA4University of Oregon Eugene OR 97403 USA5Computational Sciences Laboratory (UCY-CompSci) University of Cyprus 1678 Nicosia Cyprus

Correspondence should be addressed to Damian W I Rouson damianrousonnet

Received 8 April 2014 Accepted 5 August 2014

Academic Editor Jeffrey C Carver

Copyright copy 2015 Hari Radhakrishnan et al This is an open access article distributed under the Creative Commons AttributionLicense which permits unrestricted use distribution and reproduction in any medium provided the original work is properlycited

This paper summarizes a strategy for parallelizing a legacy Fortran 77 program using the object-oriented (OO) and coarray featuresthat entered Fortran in the 2003 and 2008 standards respectively OO programming (OOP) facilitates the construction of anextensible suite of model-verification and performance tests that drive the development Coarray parallel programming facilitatesa rapid evolution from a serial application to a parallel application capable of running on multicore processors and many-coreaccelerators in shared and distributed memory We delineate 17 code modernization steps used to refactor and parallelize theprogram and study the resulting performance Our initial studies were done using the Intel Fortran compiler on a 32-core sharedmemory server Scaling behavior was very poor and profile analysis using TAU showed that the bottleneck in the performance wasdue to our implementation of a collective sequential summation procedure We were able to improve the scalability and achievenearly linear speedup by replacing the sequential summationwith a parallel binary tree algorithmWe also tested theCray compilerwhich provides its own collective summation procedure Intel provides no collective reductions With Cray the program showslinear speedup even in distributed-memory execution We anticipate similar results with other compilers once they support thenew collective procedures proposed for Fortran 2015

1 Introduction

Background Legacy software is old software that serves a use-ful purpose In high-performance computing (HPC) a codebecomes ldquooldrdquo when it no longer effectively exploits currenthardware With the proliferation of multicore processors andmany-core accelerators onemight reasonably label any serialcode as ldquolegacy softwarerdquo The software that has proved itsutility over many years however typically has earned thetrust of its user community

Any successful strategy for modernizing legacy codesmust honor that trust This paper presents two strategiesfor parallelizing a legacy Fortran code while bolstering trust

in the result (1) a test-driven approach that verifies thenumerical results and the performance relative to the originalcode and (2) an evolutionary approach that leavesmuch of theoriginal code intact while offering a clear path to executionon multicore and many-core architectures in shared anddistributed memory

The literature on modernizing legacy Fortran codesfocuses on programmability issues such as increasing typesafety and modularization while reducing data dependanciesvia encapsulation and information hiding Achee and Carver[1] examined object extraction which involves identifyingcandidate objects by analyzing the data flow in Fortran 77code They define a cohesion metric that they use to group

Hindawi Publishing CorporationScientific ProgrammingVolume 2015 Article ID 904983 12 pageshttpdxdoiorg1011552015904983

2 Scientific Programming

global variables and parameters They then extracted meth-ods from the source code In a 1500-line code for examplethey extract 26 candidate objects

Norton and Decyk [2] on the other hand focusedon wrapping legacy Fortran with more modern inter-faces They then wrap the modernized interfaces inside anobjectabstraction layer They outline a step-by-step processthat ensures standards compliance eliminates undesirablefeatures creates interfaces adds new capabilities and thengroups related abstractions into classes and componentsExamples of undesirable features include common blockswhich potentially facilitate global data-sharing and aliasingof variable names and types In Fortran giving proceduresexplicit interfaces facilitates compiler checks on argumenttype kind and rank New capabilities they introducedincluded dynamic memory allocation

Greenough and Worth [3] surveyed tools that enhancesoftware quality by helping to detect errors and to highlightpoor practices The appendices of their report provide exten-sive summaries of the tools available from eight vendors witha very wide range of capabilities A sample of these capabili-ties includes memory leak detection automatic vectorizationand parallelization dependency analysis call-graph genera-tion and static (compile-time) as well as dynamic (run-time)correctness checking

Each of the aforementioned studies explored how toupdate codes to the Fortran 9095 standards None of thestudies explored subsequent standards and most did notemphasize performance improvement as a main goal Onerecent study however applied automated code transforma-tions in preparation for possible shared-memory loop-levelparallelization with OpenMP [4] We are aware of no pub-lished studies on employing the Fortran 2008 coarray parallelprogramming to refactor a serial Fortran 77 application Sucha refactoring for parallelization purposes is the central aim ofthe current paper

Case Study PRM Most commercial software models forturbulent flow in engineering devices solve the Reynolds-averaged Navier-Stokes (RANS) partial differential equa-tions Deriving these equations involves decomposing thefluid velocity field u into a mean part u and a fluctuatingpart u1015840

u equiv u + u1015840 (1)

Substituting (1) into amomentumbalance and then averagingover an ensemble of turbulent flows yield the following RANSequation

120588119906119895

120597119906119894

120597119909119895

= 120588119891119894+

120597

120597119909119895

[minus119901120575119894119895+ 120583(

120597119906119894

120597119909119895

+

120597119906119895

120597119909119894

) minus 1205881199061015840

1198941199061015840

119895]

(2)

where 120583 is the fluidrsquos dynamic viscosity 120588 is the fluidrsquos density119905 is the time coordinate 119906

119894and 119906119895are the 119894th and 119895th cartesian

components of u and 119909119894and 119909

119895are the 119894th and 119895th cartesian

components of the spatial coordinate x

1

2

3

1

2

3

4

5

6

7

8

9

10

Band number

Figure 1 Distribution of particles in bands in one octant

The termminus1205881199061015840

1198941199061015840

119895in (2) is called theReynolds stress tensor

Its presence poses the chief difficulty at the heart of Reynolds-averaged turbulence modeling closing the RANS equationsrequires postulating relations between the Reynolds stressand other terms appearing in the RANS equations typicallythe velocity gradient 120597119906

119895120597119909119894and scalars representing the

turbulence scale Doing so in the most common ways workswell for predicting turbulent flows in which the statistics of u1015840stay in near-equilibrium with the flow deformations appliedvia gradients inu Traditional RANSmodelswork lesswell forflows undergoing deformations so rapid that the fluctuatingfield responds solely to the deformation without time for thenonlinear interactions with itself that are the hallmark offluid turbulence The Particle Representation Model (PRM)[5 6] addresses this shortcoming Given sufficient computingresources a software implementation of the PRM can exactlypredict the response of the fluctuating velocity field to rapiddeformations

A proprietary in-house software implementation of thePRM was developed initially at Stanford University anddevelopment continued at theUniversity of CyprusThePRMuses a set of hypothetical particles over a unit hemispheresurface The particles are distributed on each octant of thehemisphere in bands as shown in Figure 1 for ten bands Thetotal number of particles is given by

119873particles = 4⏟⏟⏟⏟⏟⏟⏟

Number of octants in hemisphere

times

119873bands times (119873bands + 1)

2⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

Number of particles in one octant

= 2 times 119873bands times (119873bands + 1)

(3)

So the computational time scales quadratically with thenumber of bands used

Each particle has a set of assigned properties that describethe characteristics of an idealized flow Assigned particleproperties include vector quantities such as velocity and

Scientific Programming 3

(a) Time = 0 seconds (b) Time = 2 seconds

(c) Time = 4 seconds (d) Time = 6 seconds

Figure 2 Results of a PRM computation The particles are colored based on their initial location The applied flow condition shear flowalong the 119910-direction causes the uniformly distributed particles to aggregate along that axis

orientation as well as scalar quantities such as pressureThuseach particle can be thought of as representing the dynamicsof a hypothetical one-dimensional (1D) one-component (1C)flow Tracking a sufficiently large number of particles andthen averaging the properties of all the particles (as shownin Figure 2) that is all the possible flows considered yield arepresentation of the 3D behavior in an actual flowing fluid

Historically a key disadvantage of the PRM has beencostly execution times because a very large number ofparticles are needed to accurately capture the physics ofthe flow Parallelization can reduce this cost significantlyPrevious attempts to develop a parallel implementation of thePRM using MPI were abandoned because the developmentvalidation and verification times did not justify the gainsCoarrays allowed us to parallelize the software with minimalinvasiveness and the OO test suite facilitated a continuousbuild-and-test cycle that reduced the development time

2 Methodology

21 Modernization Strategy Test-Driven Development(TDD) grew out of the Extreme Programming movementof the 1990s although the basic concepts date as far back asthe NASA space program in the 1960s TDD iterates quicklytoward software solutions by first writing tests that specifywhat the working software must do and then writing onlya sufficient amount of application code in order to passthe test In the current context TDD serves the purpose ofensuring that our refactoring exercise preserves the expectedresults for representative production runs

Table 1 lists 17 steps employed in refactoring and par-allelizing the serial implementation of the PRM They havebeen broken down into groups that addressed various facetsof the refactoring processThe open-source CTest frameworkthat is part of CMake was used for building the tests Our firststep therefore was to construct a CMake infrastructure thatwe used for automated building and testing and to set up acode repository for version control and coordination

The next six steps address Fortran 77 features that havebeen declared obsolete in more recent standards or havebeen deprecated in the Fortran literature We did not replacecontinue statements with end do statements as these did notaffect the functionality of the code

The next two steps were crucial in setting up the buildtesting infrastructure We automated the initialization byreplacing the keyboard inputs with default values The nextstep was to construct extensible tests based on these defaultvalues which are described in Section 3

The next three steps expose optimization opportunities tothe compiler One exploits Fortranrsquos array syntax Two exploitFortranrsquos facility for explicitly declaring a procedure to beldquopurerdquo that is free of side effects including inputoutputmodifying arguments halting execution or modifying non-local state Other steps address type safety and memorymanagement

Array syntax gives the compiler a high-level view ofoperations on arrays in ways the compiler can exploit withvarious optimizations including vectorization The abilityto communicate functional purity to compilers also enablesnumerous compiler optimizations including parallelism


Table 1 Modernization steps horizontal lines indicate partial ordering

Step Details1 Set up automated builds via CMake1 and version control via Git22 Convert fixed- to free-source format via ldquoconvertf90rdquo by Metcalf33 Replace goto with do while for main loop termination4 Enforce typekindrank consistency of arguments and return values by wrapping all procedures in amodule5 Eliminate implicit typing6 Replace data statements with parameter statements7 Replace write-access to common blocks with module variables8 Replace keyboard input with default initializations9 Set up automated extensible tests for accuracy and performance via OOP and CTest110 Make all procedures outside of the main program pure11 Eliminate actualdummy array shape inconsistencies by passing array subsections to assumed-shape arrays12 Replace static memory allocation with dynamic allocation13 Replace loops with array assignments14 Expose greater parallelism by unrolling the nested loops in the particle set-up15 Balance the work distribution by spreading particles across images during set-up16 Exploit a Fortran 2015 collective procedure to gather statistics17 Study and tune performance with TAU41httpwwwcmakeorg2httpgit-scmcom3ftpftpnumericalrlacukpubMandRconvertf904httptauuoregonedu

The final steps directly address parallelism and optimiza-tion One unrolls a loop to provide for more fine-graineddata distribution The other exploits the co sum intrinsiccollective procedure that is expected to be part of Fortran 2015and is already supported by the Cray Fortran compiler (Withthe Intel compiler we write our own co sum procedure)Thefinal step involves performance analysis using the Tuning andAnalysis Utilities [7]

3 Extensible OO Test Suite

At every step we ran a suite of accuracy tests to verify that theresults of a representative simulation did not deviate from theserial codersquos results by more than 50 parts per million (ppm)We also ran a performance test to ensure that the single-imageruntime of the parallel code did not exceed the serial codersquosruntime by more than 20 (We allowed for some increasewith the expectation that significant speedup would resultfrom running multiple images)

Our accuracy tests examine tensor statistics that arecalculated using the PRM In order to establish a uniformprotocol for running tests we defined an abstract base tensorclass as shown in Listing 1

The base class provided the bindings for comparingtensor statistics displaying test results to the user andexception handling Specific tests take the form of three childclasses reynolds stress dimensionality and circulicity thatextend the tensor class and thereby inherit a responsibilityto implement the tensorrsquos deferred bindings compute resultsand expected results The class diagram is shown in Figure 3The tests then take the form

if (not stess tensorverify result (when)) amperror stop lsquoTest failedrsquo

where stress tensor is an instance of one of the three childclasses shown in Figure 3 that extend tensor ldquowhenrdquo is aninteger time stamp error stop halts all images and printsthe shown string to standard error and verify result isthe pure function shown in Listing 1 that invokes the twoaforementioned deferred bindings to compare the computedresults to the expected results

4 Coarray Parallelization

Modern HPC software must be executed on multicore pro-cessors or many-core accelerators in shared or distributedmemory Fortran provides for such flexibility by defining apartitioned global address space (PGAS) without referencinghow to map coarray code onto a particular architectureCoarray Fortran is based on the Single Program MultipleData (SPMD) model and each replication of the program iscalled an image [8] Fortran 2008 compilersmap these imagesto an underlying transport network of the compilerrsquos choiceFor example the Intel compiler uses MPI for the transportnetwork whereas the Cray compiler uses a dedicated trans-port layer

A coarray declaration of the form

real allocatable a ( ) []

facilitates indexing into the variable ldquoardquo along three regulardimensions and one codimension so

a (1 1 1) = a (1 1 1) [2]


verify_result(logical)

computed_results(tensor)expected_results(tensor)

reynolds_stress real[6]computed_results(reynolds_stress)

expected_results(reynolds_stress)

dimensionality real[6]computed_results(dimensionality)

expected_results(dimensionality)

circulicity real[6]computed_results(circulicity)

expected_results(circulicity)

reynolds_stress dimensionality circulicity

tests_passed(logical[lowast])

Figure 3 Class diagram of the testing framework The deferred bindings are shown in italics and the abstract class is shown in bold italics

module abstract tensor classtype abstract tensorcontains

procedure(return computed results) deferred ampcomputed results

procedure(return expected results) deferred ampexpected results

procedure verify resultend typeabstract interfacepure function return computed results(this) amp

result(computed values)import tensorclass(tensor) intent(in) thisreal allocatable computed values()

end function return expected results interface omitted

end abstract interfacecontainspure function verify result(this) ampresult(all tests passed)class(tensor) intent(in) thislogical all tests passedall tests passed = all(tests passed( ampthiscomputed results() thisexpected results()))

end functionend module

Listing 1 Base tensor class

copies the first element of image 2 to the first element ofwhatever image executes this line The ability to omit thecoindex on the left-hand side (LHS) played a pivotal rolein refactoring the serial code with minimal work althoughwe added codimensions to existing variablesrsquo declarationssubsequent accesses to those variables remained unmodifiedexcept where communication across images is desiredWhen

l = 0 Global particle numberdo k = 1 nb Loop over the bands

dom = 1 k Loop over the particles in band First octantl = l + 1 Do some computations Second octantl = l + 1 Do some computations Third octantl = l + 1 Do some computations Fourth octantl = l + 1 Do some computations

end doend do

Listing 2 Legacy particle loop

necessary adding coindices facilitated the construction ofcollective procedures to compute statistics

In the legacy version the computations of the particleproperties were done using two nested loops as shownin Listing 2

Distributing the particles across the images and executingthe computations inside these loops can speed up the execu-tion time This can be achieved in two ways

Method 1 works with the particles directly splitting themas evenly as possible across all the images allowing imageboundaries to occur in themiddle of a bandThis distributionis shown in Figure 4(a) To achieve this distribution the twonested do loops are replaced by one loop over the particlesand the indices for the two original loops are computed fromthe global particle number as shown in Listing 3 Howeverin this case the code becomes complex and sensitive toprecision


1

2

3

1

2

3

4

5

6

7

8

9

10

Band number

(a) Partitioning of the particles to achieve even distribution ofparticles

1

2

3

1

2

3

4

5

6

7

8

9

10

Band number

(b) Partitioning of the bands to achieve nearly even distributionof particles

Figure 4 Two different partitioning schemes were tried for load balancing

Loop over the particlesdo l = my first particle my last particle 4

k = nint(sqrt(real(l) lowast 05))m = (l minus (1 + 2 lowast k lowast (k minus 1) minus 4))4

First octant Do some computations Second octant Do some computations Third octant Do some computations Fourth octant Do some computations

end do

Listing 3 Parallel loop by splitting particles

Method 2 works with the bands splitting them across theimages to make the particle distribution as even as possibleThis partitioning is shown in Figure 4(b)Method 2 as shownin Listing 4 requires fewer changes to the original codeshown in Listing 2 but is suboptimal in load balancing

5 Results

51 Source Code Impact We applied our strategy to two serialsoftware implementations of the PRM For one version theresulting code was 10 longer than the original 639 linesversus 580 lines with no test suite In the second versionthe code expanded 40 from 903 lines to 1260 lines notincluding new inputoutput (IO) code and the test codedescribed in Section 3 The test and IO code occupiedadditional 569 lines

52 Ease of Use Coarrays versus MPI The ability to drop thecoindex from the notation as explained in Section 4 was a big

Loop over the bandsdo k = my first band my last band

Global number of last particle in (k minus 1) bandl = k lowastlowast 2 + (k minus 1) lowastlowast 2 minus 1 Loop over the particles in banddom = 1 k

First octantl = l + 1 Do some computations Second octantl = l + 1 Do some computations Third octantl = l + 1 Do some computations Fourth octantl = l + 1 Do some computations

end doend do

Listing 4 Parallel loop by splitting bands

help in parallelizing the program without making significantchanges to the source code A lot of the bookkeeping ishandled behind the scenes by the compiler making it possibleto make the parallelization more abstract but also easier tofollow For example Listing 5 shows the MPI calls necessaryto gather the local arrays into a global array on all theprocessors

The equivalent calls using the coarray syntax is the listingshown in Listing 6

Reducing the complexity of the code also reduces thechances of bugs in the code In the legacy code the arrays


integer my rank num procsinteger allocatable dimension() amp

my first my last counts displscallmpi comm size(MPI COMM WORLD num procs ierr)callmpi comm rank(MPI COMM WORLD my rank ierr)allocate(my first(num procs) my last(num procs) amp

counts(num procs) displs(num procs))my first(my rank + 1) = lbound(sn 2)my last(my rank + 1) = ubound(sn 2)callmpi allgather(MPI IN PLACE 1 MPI INTEGER ampmy first 1 MPI INTEGER MPI COMM WORLD ierr)

callmpi allgather(MPI IN PLACE 1 MPI INTEGER ampmy last 1 MPI INTEGER MPI COMM WORLD ierr)

do i = 1 num procsdispls(i) = my first(i) minus 1counts(i) = my last(i) minusmy first(i) + 1

end docallmpi allgatherv(sn 5 lowast counts(my rank + 1) ampMPI DOUBLE PRECISION sn global 5 lowast counts amp5 lowast displs MPI DOUBLE PRECISION MPI COMM WORLD ierr)

callmpi allgatherv(cr 5 lowast counts(my rank + 1) ampMPI DOUBLE PRECISION cr global 5 lowast counts amp5 lowast displs MPI DOUBLE PRECISION MPI COMM WORLD ierr)

Listing 5 Using MPI ALLGATHER to collect local arrays into a global array

integer my first[lowast] my last[lowast]my first = lbound(sn 2)my last = ubound(sn 2)do l = 1 num images()

cr global( my first[l]my last[l]) = cr()[l]sn global( my first[l]my last[l]) = sn()[l]

end do

Listing 6 Coarray method of gathering arrays

119904119899 and 119888119903 carried the information about the state of theparticles By using the coarray syntax and dropping thecoindex we were able to reuse all the original algorithmsthat implemented the core logic of the PRM This made itsignificantly easier to ensure that the refactoring did notalter the results of the model The main changes were to addcodimensions to the 119904119899 and 119888119903 declarations and update themwhen needed as shown in Listing 6

53 Scalability We intend for PRM to serve as an alternativeto turbulence models used in routine engineering designof fluid devices There is no significant difference in thePRM results when more than 1024 bands (approximately 21million particles) are used to represent the flow state so thiswas chosen as the upper limit of the size of our data setMost engineers and designers run simulations on desktopcomputers As such the upper bound on what is commonly

available is roughly 32 to 48 cores on two or four centralprocessing units (CPUs) plus additional cores on one ormoreaccelerators We also looked at the scaling performance ofparallel implementation of the PRMusingCray hardware andFortran compiler which has excellent support for distributed-memory execution of coarray programs

Figure 5 shows the speedup obtained for 200 and 400bands with the Intel Fortran compiler using the two particle-distribution schemes described in the Coarray Parallelizationsection The runs were done using up to 32 cores on the ldquofatrdquonodes of ACISS (httpaciss-computinguoregonedu) Eachnode has four Intel X7560 227GHz 8-core CPUs and 384GBof DDR3 memory We see that the speedup was very poorwhen the number of processors was increased

WeusedTAU [7] to profile the parallel runs to understandthe bottlenecks during execution Figure 6 shows the TAUplot for the runtime share for the dominant procedures usingdifferent number of images Figure 7 shows the runtimes forthe different functions on the different images The heightsof the columns show the runtime for different functions onthe individual cores There is no significant difference in theheights of the columns proving that the load balancing is verygood across the images We achieved this by mainly usingthe one-sided communication protocols of CAF as shown inListing 6 and restricting the sync statements to the collectiveprocedures as shown in Listings 7 and 8 Looking at theruntimes in Figure 6 we identified the chief bottlenecks tobe the two collective co sum procedures which sum valuesacross a coarray by sequentially polling each image for itsportion of the coarray The time required for this procedure

8 Scientific ProgrammingSp

eedu

p (r

elat

ive t

o sin

gle i

mag

e)

Number of coarray images

32

16

8

4

2

1

1 2 4 8 16 32

Split bands 200Split bands 400

Split particles 200Split particles 400

Figure 5 Speedup obtained with sequential co sum implementa-tion using multiple images on a single server

subroutine vector co sum serial(vector)real(rkind) intent(inout) vector()[lowast]integer imagesync allif (this image() == 1) then

do image = 2 num images()vector()[1] = vector()[1] + vector()[image]

end doend ifsync allif (this image() = 1) vector() = vector()[1]sync all

end subroutine

Listing 7 Unoptimized collective sum routine

is 119874(119873images) The unoptimized co sum routine for addinga vector across all images is shown in Listing 7 There is anequivalent subroutine for summing a matrix also

Designing an optimal co sum algorithm is a platform-dependent exercise best left to compilers The Fortran stan-dards committee is working on a co sum intrinsic procedurethat will likely become part of Fortran 2015 But to improvethe parallel performance of the program we rewrote thecollective co sum procedures using a binomial tree algorithmthat is 119874(log119873images) in time The optimized version of theco sum version is shown in Listing 8

The speedup obtained with the optimized co sum routineis shown in Figure 8 We see that the scaling performance ofthe program becomes nearly linear with the implementationof the optimized co sum routine We also see that the scaling

subroutine vector co sum parallel(vector)real(rkind) intent(inout) vector()[lowast]real(rkind) allocatable temp()integer image stepallocate (temp mold = vector)step = 2do while (step2 lt= num images())sync allif (this image() + step2 lt= num images()) thentemp = vector + vector[this image() + step2]

elsetemp = vector

end ifsync allvector = tempstep = step lowast 2

end dosync allif (this image() = 1) vector = vector[1]sync all

end subroutine

Listing 8 Optimized collective sum routine

efficiency increases when the problem size is increased Thisindicates that the poor scaling at smaller problem sizes is dueto communication and synchronization [9]

The TAU profile analysis of the runs using differentnumber of images is shown in Figure 9While there is a smallincrease in the co sumcomputation timewhen increasing thenumber of images it is significantly lower than increase intime for the unoptimized version

To fully understand the impact of the co sum routines wealso benchmarked the program using the Cray compiler andhardware Cray has native support for the co sum directive inthe compiler Cray also uses its own communication libraryonCray hardware instead of building on top ofMPI as is doneby the Intel compiler As we can see in Figure 10 the parallelcode showed very good strong scaling on the Cray hardwareup to 128 images for the problem sizes that we tested

We also looked at the TAU profiles of the parallel code onthe Cray hardware shown in Figure 11 The profile analysisshows that the time is spent mainly in the time advancementloop when the native co sum implementation is used

We hope that with the development and implementationof intrinsic co sum routines as part of the 2015 Fortranstandard the Intel compiler will also improve its strongscaling performance with larger number of images Table 2shows the raw runtimes for the different runs using 128 bandswhose TAU profiles have been shown in Figures 6 9 and 11The runtimes for one to four images are very close but theyquickly diverge as we increase the number of images due tothe impact of the collective procedures

Table 3 shows the weak scaling performance of theprogram using the optimized co sum procedures using theIntel compiler The number of particles as shown in Figure 1scales as the square of the number of bands Therefore when


Metric timeValue exclusiveUnits seconds

TAU application =gt matrix_co_sum_serial

matrix_co_sum_serial

TAU application

TAU application =gt vector_co_sum_serial

vector_co_sum_serial

0001

0001

0539 (37219358)

0539 (37219358)

1453 (100267616)

1453 (100267616)

3578 (246922783)

3578 (246922783)

11753 (811110361)

11753 (811110361)

47596 (3284726756)

47596 (3284726756)

36375

18779 (51626)9794 (26926)

5371 (14766)2472 (6797)1695 (4661)

0004

0004

0959 (21948832)

0959 (21948832)

1016 (23261172)

1016 (23261172)

1674 (38315516)

1674 (38315516)

4399 (100706323)

4399 (100706323)27483 (629190839)

27483 (629190839)

intel_iprm_serial1ppk - Meanintel_iprm_serial2ppk - Meanintel_iprm_serial4ppk - Mean


Figure 6 TAUprofiling analysis of function runtimeswhen using the unoptimized co sum routines with 1 2 4 8 16 and 32 imagesThe TAUapplication is the main program wrapped by TAU for profiling and TAU application =gt refers to functions wrapped by TAU This notationis also seen in Figures 7 and 9

Figure 7 TAU analysis of load balancing and bottlenecks for the parallel code using 32 images


Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands

Figure 8 Speedup obtained with parallel co sum implementation using multiple images on a single server


intel_iprm_parallel1ppk - Meanintel_iprm_parallel2ppk - Meanintel_iprm_parallel4ppk - Mean


TAU application

TAU application =gt vector_co_sum_parallel

vector_co_sum_parallel

TAU application =gt matrix_co_sum_parallel

TAU application =gt accumulate_2nd_moments

matrix_co_sum_parallel

38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)

Figure 9 TAU profiling analysis of function runtimes when using the optimized co sum routines with 1 2 4 8 16 and 32 images


Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128

Figure 10 Speedup obtained with parallel co sum implementation using multiple images on a distributed-memory Cray cluster


cray_1ppk - Meancray_2ppk - Meancray_4ppk - Mean


main_

adv$setup_and_advance_module_

accumulate_2nd_moments$output_and_statistics_module_

896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)

Figure 11 TAU profiling analysis of function runtimes when using the Cray native co sum routines with 1 2 4 8 16 and 32 images

doubling the number of bands the number of processorsmust be quadrupled to have the same execution time Thescaling efficiency for the larger problem drops because ofmemory requirements the objects fit in the heap and mustbe swapped out as needed increasing the execution time

6 Conclusions and Future Work

We demonstrated a strategy for parallelizing legacy Fortran77 codes using Fortran 2008 coarraysThe strategy starts withconstructing extensible tests using Fortranrsquos OOP featuresThe tests check for regressions in accuracy and performanceIn the PRM case study our strategy expanded two Fortran

77 codes by 10 and 40 exclusive of the test and IOinfrastructure The most significant code revision involvedunrolling two nested loops that distribute particles acrossimagesThe resulting parallel code achieves even load balanc-ing but poor scaling TAU identified the chief bottleneck as asequential summation scheme

Based on these preliminary results we rewrote ourco sum procedure and the speedup showed markedimprovement We also benchmarked the native co sumimplementation available in the Cray compiler Our resultsshow that the natively supported collective procedures showthe best scaling performance even when using distributedmemory We hope that future native support for collective


Table 2 Runtime in seconds for parallel using 128 bands and different collective sum routines

Number of Images1 2 4 8 16 32

Intel Serial co sum 3555 1980 1169 973 1871 6682Intel Parallel co sum 3730 1933 1000 617 462 541Cray Native co sum 4671 2368 1188 606 306 173

Table 3 Weak scaling performance of coarray version

Number of images Number of bands Number of particles Particles per image Time in seconds Runtime per particle Efficiency1 128 33024 33024 44279 134 times 10

minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777

procedures in Fortran 2015 by all the compilers will bringsuch performance to all platforms

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

The initial code refactoring was performed at the Uni-versity of Cyprus with funding from the European Com-mission Marie Curie ToK-DEV grant (Contract MTKD-CT-2004-014199) Part of this work was also supported bythe Cyprus Research Promotion Foundationrsquos FrameworkProgramme for Research Technological Development andInnovation 2009-2010 (ΔEΣMH 2009-2010) under GrantTΠEΠΛHPO0609(BE)11 This work used resources ofthe National Energy Research Scientific Computing Cen-ter which is supported by the Office of Science of theUS Department of Energy under Contract no DE-AC02-05CH11231 This work also used hardware resources fromthe ACISS cluster at the University of Oregon acquired bya Major Research Instrumentation grant from the NationalScience Foundation Office of Cyber Infrastructure ldquoMRI-R2 Acquisition of an Applied Computational Instrument forScientific Synthesis (ACISS)rdquo Grant no OCI-0960354 Thisresearch was also supported by Sandia National Laboratoriesa multiprogram laboratory operated by Sandia Corporationa LockheedMartin Company for the National Nuclear Secu-rity Administration under Contract DE-AC04-94-AL85000Portions of the Sandia contribution to this work were fundedby the New Mexico Small Business Administration and theOffice of Naval Research

References

[1] B L Achee and D L Carver ldquoCreating object-oriented designsfrom legacy FORTRAN coderdquo Journal of Systems and Softwarevol 39 no 2 pp 179ndash194 1997

[2] C D Norton and V K Decyk ldquoModernizing Fortran 77 legacycodesrdquo NASA Tech Briefs vol 27 no 9 p 72 2003

[3] C Greenough and D J Worth ldquoThe transformation of legacysoftware some tools and processesrdquo Tech Rep TR-2004-012Council for the Central Laboratory of the Research CouncilsRutherford Appleton Laboratories Oxfordshire UK 2004

[4] F G Tinetti and M Mendez ldquoFortran Legacy software sourcecode update and possible parallelisation issuesrdquoACMSIGPLANFortran Forum vol 31 no 1 pp 5ndash22 2012

[5] S C Kassinos and W C Reynolds ldquoA particle representa-tion model for the deformation of homogeneous turbulencerdquoin Annual Research Briefs pp 31ndash61 Center for TurbulenceResearch Stanford University Stanford Calif USA 1996

[6] S C Kassinos and E Akylas ldquoAdvances in particle rep-resentation modeling of homogeneous turbulence from thelinear PRM version to the interacting viscoelastic IPRMrdquo inNew Approaches in Modeling Multiphase Flows and Dispersionin Turbulence Fractal Methods and Synthetic Turbulence FNicolleau C Cambon J-M Redondo J Vassilicos M Reeksand A Nowakowski Eds vol 18 of ERCOFTAC Series pp 81ndash101 Springer Dordrecht The Netherlands 2012

[7] S S Shende and A D Malony ldquoThe TAU parallel performancesystemrdquo International Journal of High Performance ComputingApplications vol 20 no 2 pp 287ndash311 2006

[8] M Metcalf J K Reid and M Cohen Modern FortranExplained Oxford University Press 2011

[9] H Radhakrishnan D W I Rouson K Morris S Shendeand S C Kassinos ldquoTest-driven coarray parallelization of alegacy Fortran applicationrdquo in Proceedings of the 1st Interna-tional Workshop on Software Engineering for High PerformanceComputing in Computational Science and Engineering pp 33ndash40 ACM November 2013

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014


Distributed Sensor Networks


Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014


ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014


Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014


Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications


Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia


Biomedical Imaging


ArtificialNeural Systems

Advances in


RoboticsJournal of



Computational Intelligence and Neuroscience

Industrial EngineeringJournal of


Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014


Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in



global variables and parameters They then extracted meth-ods from the source code In a 1500-line code for examplethey extract 26 candidate objects

Norton and Decyk [2] on the other hand focusedon wrapping legacy Fortran with more modern inter-faces They then wrap the modernized interfaces inside anobjectabstraction layer They outline a step-by-step processthat ensures standards compliance eliminates undesirablefeatures creates interfaces adds new capabilities and thengroups related abstractions into classes and componentsExamples of undesirable features include common blockswhich potentially facilitate global data-sharing and aliasingof variable names and types In Fortran giving proceduresexplicit interfaces facilitates compiler checks on argumenttype kind and rank New capabilities they introducedincluded dynamic memory allocation

Greenough and Worth [3] surveyed tools that enhancesoftware quality by helping to detect errors and to highlightpoor practices The appendices of their report provide exten-sive summaries of the tools available from eight vendors witha very wide range of capabilities A sample of these capabili-ties includes memory leak detection automatic vectorizationand parallelization dependency analysis call-graph genera-tion and static (compile-time) as well as dynamic (run-time)correctness checking

Each of the aforementioned studies explored how toupdate codes to the Fortran 9095 standards None of thestudies explored subsequent standards and most did notemphasize performance improvement as a main goal Onerecent study however applied automated code transforma-tions in preparation for possible shared-memory loop-levelparallelization with OpenMP [4] We are aware of no pub-lished studies on employing the Fortran 2008 coarray parallelprogramming to refactor a serial Fortran 77 application Sucha refactoring for parallelization purposes is the central aim ofthe current paper

Case Study PRM Most commercial software models forturbulent flow in engineering devices solve the Reynolds-averaged Navier-Stokes (RANS) partial differential equa-tions Deriving these equations involves decomposing thefluid velocity field u into a mean part u and a fluctuatingpart u1015840

u equiv u + u1015840 (1)

Substituting (1) into amomentumbalance and then averagingover an ensemble of turbulent flows yield the following RANSequation

120588119906119895

120597119906119894

120597119909119895

= 120588119891119894+

120597

120597119909119895

[minus119901120575119894119895+ 120583(

120597119906119894

120597119909119895

+

120597119906119895

120597119909119894

) minus 1205881199061015840

1198941199061015840

119895]

(2)

where 120583 is the fluidrsquos dynamic viscosity 120588 is the fluidrsquos density119905 is the time coordinate 119906

119894and 119906119895are the 119894th and 119895th cartesian

components of u and 119909119894and 119909

119895are the 119894th and 119895th cartesian

components of the spatial coordinate x

1

2

3

1

2

3

4

5

6

7

8

9

10

Band number

Figure 1 Distribution of particles in bands in one octant

The termminus1205881199061015840

1198941199061015840

119895in (2) is called theReynolds stress tensor

Its presence poses the chief difficulty at the heart of Reynolds-averaged turbulence modeling closing the RANS equationsrequires postulating relations between the Reynolds stressand other terms appearing in the RANS equations typicallythe velocity gradient 120597119906

119895120597119909119894and scalars representing the

turbulence scale Doing so in the most common ways workswell for predicting turbulent flows in which the statistics of u1015840stay in near-equilibrium with the flow deformations appliedvia gradients inu Traditional RANSmodelswork lesswell forflows undergoing deformations so rapid that the fluctuatingfield responds solely to the deformation without time for thenonlinear interactions with itself that are the hallmark offluid turbulence The Particle Representation Model (PRM)[5 6] addresses this shortcoming Given sufficient computingresources a software implementation of the PRM can exactlypredict the response of the fluctuating velocity field to rapiddeformations

A proprietary in-house software implementation of thePRM was developed initially at Stanford University anddevelopment continued at theUniversity of CyprusThePRMuses a set of hypothetical particles over a unit hemispheresurface The particles are distributed on each octant of thehemisphere in bands as shown in Figure 1 for ten bands Thetotal number of particles is given by

119873particles = 4⏟⏟⏟⏟⏟⏟⏟

Number of octants in hemisphere

times

119873bands times (119873bands + 1)

2⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

Number of particles in one octant

= 2 times 119873bands times (119873bands + 1)

(3)

So the computational time scales quadratically with thenumber of bands used

Each particle has a set of assigned properties that describethe characteristics of an idealized flow Assigned particleproperties include vector quantities such as velocity and







2 Methodology






















a (1 1 1) = a (1 1 1) [2]

























end doend do







1

2

3

1

2

3

4

5

6

7

8

9

10

Band number


1

2

3

1

2

3

4

5

6

7

8

9

10

Band number






end do



5 Results






end doend do
















end do








eedu

p (r

elat

ive t

o sin

gle i

mag

e)


32

16

8

4

2

1

1 2 4 8 16 32







end subroutine






elsetemp = vector



end subroutine












TAU application



0001

0001

0539 (37219358)

0539 (37219358)

1453 (100267616)

1453 (100267616)

3578 (246922783)

3578 (246922783)

11753 (811110361)

11753 (811110361)

47596 (3284726756)

47596 (3284726756)

36375

18779 (51626)9794 (26926)

5371 (14766)2472 (6797)1695 (4661)

0004

0004

0959 (21948832)

0959 (21948832)

1016 (23261172)

1016 (23261172)

1674 (38315516)

1674 (38315516)

4399 (100706323)

4399 (100706323)27483 (629190839)

27483 (629190839)






Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands





TAU application






38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)



Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in









2 Methodology






















a (1 1 1) = a (1 1 1) [2]

























end doend do







1

2

3

1

2

3

4

5

6

7

8

9

10

Band number


1

2

3

1

2

3

4

5

6

7

8

9

10

Band number






end do



5 Results






end doend do
















end do








eedu

p (r

elat

ive t

o sin

gle i

mag

e)


32

16

8

4

2

1

1 2 4 8 16 32







end subroutine






elsetemp = vector



end subroutine












TAU application



0001

0001

0539 (37219358)

0539 (37219358)

1453 (100267616)

1453 (100267616)

3578 (246922783)

3578 (246922783)

11753 (811110361)

11753 (811110361)

47596 (3284726756)

47596 (3284726756)

36375

18779 (51626)9794 (26926)

5371 (14766)2472 (6797)1695 (4661)

0004

0004

0959 (21948832)

0959 (21948832)

1016 (23261172)

1016 (23261172)

1674 (38315516)

1674 (38315516)

4399 (100706323)

4399 (100706323)27483 (629190839)

27483 (629190839)






Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands





TAU application






38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)



Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in


















a (1 1 1) = a (1 1 1) [2]

























end doend do







1

2

3

1

2

3

4

5

6

7

8

9

10

Band number


1

2

3

1

2

3

4

5

6

7

8

9

10

Band number






end do



5 Results






end doend do
















end do








eedu

p (r

elat

ive t

o sin

gle i

mag

e)


32

16

8

4

2

1

1 2 4 8 16 32







end subroutine






elsetemp = vector



end subroutine












TAU application



0001

0001

0539 (37219358)

0539 (37219358)

1453 (100267616)

1453 (100267616)

3578 (246922783)

3578 (246922783)

11753 (811110361)

11753 (811110361)

47596 (3284726756)

47596 (3284726756)

36375

18779 (51626)9794 (26926)

5371 (14766)2472 (6797)1695 (4661)

0004

0004

0959 (21948832)

0959 (21948832)

1016 (23261172)

1016 (23261172)

1674 (38315516)

1674 (38315516)

4399 (100706323)

4399 (100706323)27483 (629190839)

27483 (629190839)






Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands





TAU application






38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)



Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in



























end doend do







1

2

3

1

2

3

4

5

6

7

8

9

10

Band number


1

2

3

1

2

3

4

5

6

7

8

9

10

Band number






end do



5 Results






end doend do
















end do








eedu

p (r

elat

ive t

o sin

gle i

mag

e)


32

16

8

4

2

1

1 2 4 8 16 32







end subroutine






elsetemp = vector



end subroutine












TAU application



0001

0001

0539 (37219358)

0539 (37219358)

1453 (100267616)

1453 (100267616)

3578 (246922783)

3578 (246922783)

11753 (811110361)

11753 (811110361)

47596 (3284726756)

47596 (3284726756)

36375

18779 (51626)9794 (26926)

5371 (14766)2472 (6797)1695 (4661)

0004

0004

0959 (21948832)

0959 (21948832)

1016 (23261172)

1016 (23261172)

1674 (38315516)

1674 (38315516)

4399 (100706323)

4399 (100706323)27483 (629190839)

27483 (629190839)






Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands





TAU application






38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)



Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in




1

2

3

1

2

3

4

5

6

7

8

9

10

Band number


1

2

3

1

2

3

4

5

6

7

8

9

10

Band number






end do



5 Results






end doend do
















end do








eedu

p (r

elat

ive t

o sin

gle i

mag

e)


32

16

8

4

2

1

1 2 4 8 16 32







end subroutine






elsetemp = vector



end subroutine












TAU application



0001

0001

0539 (37219358)

0539 (37219358)

1453 (100267616)

1453 (100267616)

3578 (246922783)

3578 (246922783)

11753 (811110361)

11753 (811110361)

47596 (3284726756)

47596 (3284726756)

36375

18779 (51626)9794 (26926)

5371 (14766)2472 (6797)1695 (4661)

0004

0004

0959 (21948832)

0959 (21948832)

1016 (23261172)

1016 (23261172)

1674 (38315516)

1674 (38315516)

4399 (100706323)

4399 (100706323)27483 (629190839)

27483 (629190839)






Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands





TAU application






38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)



Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in














end do








eedu

p (r

elat

ive t

o sin

gle i

mag

e)


32

16

8

4

2

1

1 2 4 8 16 32







end subroutine






elsetemp = vector



end subroutine












TAU application



0001

0001

0539 (37219358)

0539 (37219358)

1453 (100267616)

1453 (100267616)

3578 (246922783)

3578 (246922783)

11753 (811110361)

11753 (811110361)

47596 (3284726756)

47596 (3284726756)

36375

18779 (51626)9794 (26926)

5371 (14766)2472 (6797)1695 (4661)

0004

0004

0959 (21948832)

0959 (21948832)

1016 (23261172)

1016 (23261172)

1674 (38315516)

1674 (38315516)

4399 (100706323)

4399 (100706323)27483 (629190839)

27483 (629190839)






Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands





TAU application






38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)



Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in




eedu

p (r

elat

ive t

o sin

gle i

mag

e)


32

16

8

4

2

1

1 2 4 8 16 32







end subroutine






elsetemp = vector



end subroutine












TAU application



0001

0001

0539 (37219358)

0539 (37219358)

1453 (100267616)

1453 (100267616)

3578 (246922783)

3578 (246922783)

11753 (811110361)

11753 (811110361)

47596 (3284726756)

47596 (3284726756)

36375

18779 (51626)9794 (26926)

5371 (14766)2472 (6797)1695 (4661)

0004

0004

0959 (21948832)

0959 (21948832)

1016 (23261172)

1016 (23261172)

1674 (38315516)

1674 (38315516)

4399 (100706323)

4399 (100706323)27483 (629190839)

27483 (629190839)






Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands





TAU application






38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)



Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in







TAU application



0001

0001

0539 (37219358)

0539 (37219358)

1453 (100267616)

1453 (100267616)

3578 (246922783)

3578 (246922783)

11753 (811110361)

11753 (811110361)

47596 (3284726756)

47596 (3284726756)

36375

18779 (51626)9794 (26926)

5371 (14766)2472 (6797)1695 (4661)

0004

0004

0959 (21948832)

0959 (21948832)

1016 (23261172)

1016 (23261172)

1674 (38315516)

1674 (38315516)

4399 (100706323)

4399 (100706323)27483 (629190839)

27483 (629190839)






Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands





TAU application






38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)



Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in




Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands256 bands

512 bands1024 bands





TAU application






38789

18606 (47966)9528 (24562)

5096 (13138)2403 (6195)1736 (4474)

0006

0006

0716 (11844856)

0716 (11844856)

0795 (13162583)

0795 (13162583)

0927 (15345441)

0927 (15345441)

5367 (88841014)

5367 (88841014)

0002

0002

0207 (8810425)

0207 (8810425)

0339 (14384098)

0339 (14384098)

062 (26333376)

062 (26333376)

1029 (43708323)

1029 (43708323)

3043 (129218163)

3043 (129218163)

2065

1051 (50894)0471 (22799)0283 (13687)0158 (7646)0125 (6006)

0672 (11126941)

0672 (11126941)



Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in




Spee

dup

(rel

ativ

e to

singl

e im

age)


32

16

8

4

2

1

1 2 4 8 16 32

128 bands64 bands

256 bands

64

64

128

128





main_



896719

449256 (501)224595 (25046)

112369 (12531)56027 (6248)27957 (3118)

48134

24403 (50697)12215 (25378)6263 (13012)3293 (6841)1978 (411)

31599

23662 (74882)1482 (46902)17001 (53804)16748 (53002)1347 (42629)













minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in









minus3 10004 256 131584 32896 44953 137 times 10

minus3 097816 512 525312 32832 49400 150 times 10

minus3 08932 256 131584 65792 10103 154 times 10

minus3 10008 512 525312 65664 10211 156 times 10

minus3 098732 1024 2099200 65600 12975 198 times 10

minus3 0777




Acknowledgments


References

















Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in










Advances in

FuzzySystems


Volume 2014












Journal of

Journal of





Advances in

Multimedia


Biomedical Imaging



Advances in


RoboticsJournal of










Advances in



Documents

Research Article Using Coarrays to Parallelize Legacy Fortran …downloads.hindawi.com/journals/sp/2015/904983.pdf · 2019. 7. 31. · Research Article Using Coarrays to Parallelize