Exploring Parallelism on the Intel R© Xeon PhiTM

with Lattice-QCD kernelsAndreas Diavastos∗†, Giannos Stylianou∗, Giannis Koutsou∗

∗Computation-based Science and Technology Research Centre, The Cyprus Institute{a.diavastos, g.stylianou, g.koutsou}@cyi.ac.cy

†Department of Computer Science, University of Cyprus{diavastos}@cs.ucy.ac.cy

Abstract—Progress in processor technology has allowed foraddressing scientific, compute-intensive problems, that were con-sidered unsolvable only a few decades earlier. Currently, allthe more scientific disciplines are making use of computationalmethods, increasing the demands for innovation in hardwaresuch as more power efficient and scalable computers. The use ofaccelerators is growing rapidly, since they pack a relatively largenumber of floating point operations in a low power profile, thusincreasing the Flop-to-Watt ratio.

In this work we explore several techniques for increasing thescalability of certain scientific kernels suitable for accelerators,such as the NVIDIA GPUs and the Intel Xeon Phi. For thepurposes of this study, we use compute kernels from latticeQuantum Chromodynamics (QCD) applications. GPUs have beenused in the lattice community for some time now, therefore inthis paper we use GPU results mainly as a reference point forthe newly introduced Xeon Phi. Our target is to study ways ofexploring the parallelism offered by vectorization on acceleratorswith very wide vector units.

To this end, we have implemented two kernels that derive fromthe Wilson Dslash operator, which consists the computationalhot-spot in lattice QCD applications. We see a 660% increase inbandwidth for certain parts of the application thanks to the auto-vectorization by the compiler, when compared to scalar code. Inother kernels where complex arithmetic operations dominate,our hand-vectorized code out-performs the compiler’s auto-vectorization. In this paper we find that our so-called HoppingVector-friendly Ordering allows for more efficient vectorizationof complex arithmetic floating point operations. Using this datalayout, we manage to increase the sustained bandwidth by ≈1.8x.We also present findings on various other data manipulationtechniques which improve the sustained bandwidth by ≈1.4x.

Keywords-Lattice QCD; Xeon Phi; many-cores; coprocessors,accelerators;

I. INTRODUCTION

An emerging trend in the current design of supercomputersis the use of accelerators as the main source of computingpower. This is evident by the fact that five of the top tensupercomputers [1] today use accelerators as co-processingunits of the CPU. Namely Titan [2], Piz Daint [3], and aUS government Cray CS-Storm machine [4] use NVIDIAGPUs, while Tianhe-2 [5] and Stampede [6] are equippedwith Intel Xeon Phi coprocessors. The increase in the useof such coprocessors is understood, since they allow packinga higher floating point performance within a smaller powerbudget compared to regular CPUs.

To achieve good application performance using such com-pute engines is a non-trivial task however, even for applicationprogrammers well versed in programming. The applicationprogrammer faces challenges such as porting the code toa specific programming model that supports the targetedaccelerator, taking care of load balancing between hosts andaccelerators, efficient partitioning of the problem data withina heterogeneous memory hierarchy, and explicitly managingcommunication between hosts and accelerators in a systemof thousands of nodes. Many accelerators also have specialpurpose compute units that require additional effort to makeuse of, such as CUDA cores which usually require re-factoringof compute kernels such that they are suitable for streamingfloating point engines. Another example are the wide SIMDfloating point units on the Xeon Phi that also require re-thinking of data layout to match the SIMD vector width.

In this work we focus on performance gains which canbe achieved by wide vector units on accelerator-based sys-tems and in particular systems with Xeon Phi coprocessors.The floating point units of the Xeon Phi architecture are512-bit wide vector units with a throughput of 32 single-precision or 16 double-precision floating point operations percycle per core. To efficiently use these vector units requireseither relying on the auto-vectorization by the compiler incases where this is possible, or alternatively in using so-called intrinsic functions, i.e. vector operations exposed asfunctions to the programming language. In this study, weevaluate several techniques involving reordering of data andscheduling of floating point operations, that allow auto- orhand-vectorization of the compute kernels which are to beoptimized.

For the purposes of this study we are interested in computa-tional kernels arising from the field of lattice Quantum Chro-modynamics (QCD). QCD is the part of the Standard Modelof physics which describes the strong interactions, which inturn is responsible for the forces which hold the quarks withinprotons and neutrons. Phenomena which are dominated by thestrong interaction include quark-gluon plasma, which was thestate of the very early universe and the birth, life, and deathof stars. Currently, the most well established way to obtainquantitative results from QCD is via simulation. Simulationsproceed by defining the theory on a discreet, 4-dimensional

space-time lattice, and carrying out a Marcov-Chain MonteCarlo to produce representative configurations of the strongforce fields, which are the main data sets that are analyzed toobtain the observables of interest.

Lattice QCD researchers are among the earliest adoptersof innovative computing devices. Examples include the QCDon Chip (QCDOC) project [7], a joint collaboration betweenIBM and lattice QCD researchers at Columbia University toco-design an architecture optimized for lattice QCD and theAPE machines [8], another notable example of such a co-designed architecture. More recent examples include one ofthe earliest adoption of GPUs as computing devices for use inscientific applications [9].

Lattice QCD applications have characteristics which makethem optimal pilot applications for new architectures. Theyinvolve relatively simple compute kernels, which only involvemultiplications, and additions/subtractions. Therefore the totalnumber of floating point operations is fixed and countable. Thedependencies of operations in lattice QCD kernels are alsodeterministic and repetitive, meaning communication patternsare predictable. These characteristics allow hand-optimizinglattice QCD kernels with relatively moderate human effort,which then allows for evaluating various programming modelsand parallelization strategies. For these reasons we use latticeQCD as our target application within this work, to evaluatethe various optimization strategies that follow.

Our findings show that compiler optimizations can ef-ficiently vectorize certain applications with operations thatinvolve single-precision data and achieve a significant sus-tained bandwidth increase of 660% on the Intel Xeon Phi.In other cases, such as when the compute kernels involvecomplex arithmetic, the data layout is such that the compilerappears to not produce vectorized operations. In these cases,we implement various data manipulation optimizations thatchange the layout of the data in such a way that the datacan be vectorized, either automatically by the compiler ormanually by the user. A Hopping Vector-friendly orderingtechnique, is found to achieve a sustained bandwidth increaseof ≈1.8x compared to non-vectorized code. We note that ourkernels perform a fixed number of floating point operations perbyte of I/O, meaning that sustained bandwidth is proportionalto sustained floating point performance by a constant factor,which only depends on the kernel.

The remainder of this paper is organized as follows. InSection II we present the Intel MIC architecture of the XeonPhi coprocessor, in Section III we shortly introduce the latticeQCD kernels we will use as benchmarks in this work. In Sec-tion IV we describe the proposed data ordering techniques andother data manipulation techniques for improving sustainedbandwidth. In Section V we present our evaluation platformand explain our results. In Section VI we show similarexperiences from the literature and finally in Section VII weoutline our conclusions.

Core

L2

Core

L2

GDDR MC

. . .

CoreL2

CoreL2

GDDR MC

. . .

Core

L2

Core

L2GDDR

MC

. . .

Core L2

Core L2

GDDR MC

. . .

TDTD

TD

TD

TDTD

TD

TD

Fig. 1. The Intel MIC top level architecture.

II. INTEL MIC ARCHITECTURE

The Xeon Phi coprocessor [10] (codename Knights Corner)is part of the Many Integrated Core (MIC) Architecturefamily from Intel. Each coprocessor is equipped with up to61 processor cores connected by a high-performance on-diebidirectional ring interconnect as shown in Figure 1. Eachcoprocessor includes 8 memory controllers supporting upto 16 GDDR5 channels (2 per memory controller) with atheoretical aggregate bandwidth of 352 GB/s. Each core isa fully functional, in-order core, that supports 4 hardwarethreads. To reduce hot-spot contention for data among thecores, a distributed tag directory is implemented such thatevery physical address is uniquely mapped through a reversibleone-to-one address hashing function. This also provides for aframework for more elaborate coherence protocol mechanismsthan the individual cores could provide.

The working frequency of the Xeon Phi cores is 1GHz(up to 1.3GHz) and their architecture is based on the x86Instruction Set Architecture (ISA), extended with 64-bit ad-dressing and 512-bit wide vector instructions and registers thatallow for significant increase in parallel computation. Witha 512-bit vector set, a throughput of 8 double-precision or16 single-precision floating point operations can be executedon each core per cycle. Assuming a throughput of one fusedmultiply-add operation per cycle, this brings the theoreticalpeak double precision floating point performance to almost 1Tflop/s. Each core has a 32KB L1 data cache, a 32KB L1instruction cache and a 512KB L2 cache. The L2 caches ofall cores are interconnected with each other and the memorycontrollers via a bidirectional ring bus, effectively creating ashared, cache-coherent last-level cache of up to 32MB.

Although coprocessors in general are used as acceleratorsfor offloading computation from host processes, the Xeon Phiis more flexible with the execution of applications [11]. The

Main Process

Offload Threads

Offload Threads

PCI ExpressHost MICs

Idle

Main Process

Idle

PCI ExpressHost MICs

Process 1

Process 2

Process 3

PCI Express

`

Host MICs

(a) (b) (c)

Fig. 2. Intel Xeon Phi execution modes: (a) Offload mode, (b) Coprocessor-only mode and (c) Symmetric mode. The arrows show the communicationbetween processes of the same program

Xeon Phi offers three different modes of execution: the Offloadmode, the Coprocessor-only mode and the Symmetric mode.

In Offload mode, the Host CPU executes the main processof the program while the Xeon Phi is used as an acceleratorfor offloading computation in the form of parallel threads asshown in Figure 2(a). The offloading of parallel threads to theXeon Phi is done using directives in the source code to denotethe parallel section to be executed on the coprocessor. In thismode, the Xeon Phi works the same way as a General PurposeGPU (GPGPU). Although this mode provides for a more user-friendly programming environment for the Xeon Phi, the userhas limited control over the computation that is scheduled onthe coprocessor and the communication between the host andthe coprocessor.

In Coprocessor-only mode, the Xeon Phi works as the onlycompute unit for the program while the rest of the resourcesin the node are not used (Figure 2(b)). The main process islaunched directly on the Xeon Phi and exploits the parallelismby either using multiple processes or multiple threads.

Finally, in Symmetric mode, multiple processes of the sameprogram can be launched on both the host CPU and the XeonPhi as shown in Figure 2(c). This mode works the sameway as any other message passing model. The user is solelyresponsible to handle the parallelization of the application, thescheduling of the processes and the movement of the data onthe system.

III. BENCHMARK KERNELS

Lattice Quantum Chromodynamics (LQCD) [12] is a dis-crete formulation of QCD on a finite 4-dimensional space-timelattice that enables numerical simulation of QCD using Monte-Carlo methods. LQCD is the most well established method forsolving the fundamental theory of strong interactions. Here weuse components of the so-called Wilson Dslash operator [13]as the kernels we target for oprimization. The Wilson Dslashoperator applied to a vector ψ is given in Equation 1.

∑x′D[u](x, x′)ψ(x′) =

1

2κψ(x) +

1

2

3∑µ=0

[(I − γµ)uµ(x)ψ(x+ µ)+

(I + γµ)u−1µ (x− µ)ψ(x− µ)]

(1)

where κ is a parameter of the theory, I is the 4×4 unitmatrix, γµ are the four 4×4 gamma-matrices in some basis

Fig. 4. The elements involved in the stencil operation of the 2D Laplacian

(e.g. the Dirac matrices), ψ(x) is a so-called spinor field, xand x′ are space-time (four components) coordinates, uµ(x)represents a so-called gauge field, and µ is an index forthe direction (e.g. µ = 0, 1, 2, 3 may represent the t, x, y, zdirections). The fermion fields are vectors in spinor and colorspace, meaning they have a color index which takes threevalues and a spinor index which takes four values. A spinoris defined on each lattice site. A gauge-link uµ(x) is definedbetween lattice sites, i.e. linking lattice sites, and thereforecarries a coordinate (x) and a direction index (µ). Gauge-links are color-matrices, meaning they are matrices in colorspace and so they carry two color indices. Therefore, e.g. theoperation uµ(x)ψ(x+µ) in Equation 1 implies a matrix-vectormultiplication of the 3×3 matrix u at site x in direction µ withthe three-component vector ψ which resides on the site neigh-boring x in direction µ. Similarly, the multiplication with thecombination I±γµ implies a matrix-vector multiplication overthe spinor components of ψ. Lattice QCD simulations typicallyspend 70-80% of the execution applying this operator.

Since gamma-matrices contain only one non-zero valueper column or row, applying a gamma-matrix to a spinorinvolves only re-ordering of its components and is thereforeimplemented to involve no floating point operations. Giventhis, we identify two computational components in Equation 1:a dense floating-point component, in which an array of 3×3complex matrices is multiplied to the right by an array ofspinors, and a component more demanding on memory I/Oand cache locality, which involves the data dependencies onthe nearest neighboring sites. More precisely, if one ignores themultiplication with the gamma-matrices and the gauge-links,Equation 1 becomes a 4-dimensional, nearest neighboringstencil operation.

A. Stencil operator

The first kernel we implement is a nearest-neighbor 2Dstencil operation. The idea is to emulate aspects of the datareuse required in the Wilson-Dirac equation. More precisely,the stencil we implement resembles a discreet 2D Laplacianoperation applied to a field φ given by Equation 2.

B[y+0][x+0] = a*A[y+0][x+0] + b*(A[y+0][x+1] + A[y+0][x-1] + A[y+1][x+0] + A[y-1][x+0])B[y+0][x+1] = a*A[y+0][x+1] + b*(A[y+0][x+2] + A[y+0][x+0] + A[y+1][x+1] + A[y-1][x+1])B[y+0][x+2] = a*A[y+0][x+2] + b*(A[y+0][x+3] + A[y+0][x+1] + A[y+1][x+2] + A[y-1][x+2])B[y+0][x+3] = a*A[y+0][x+3] + b*(A[y+0][x+4] + A[y+0][x+2] + A[y+1][x+3] + A[y-1][x+3])

Fig. 3. The first four iterations of the Laplacian-like operation on array A to yield the output array B

φ(x) =1

1 + 4σ

{φ(x) + σ[φ(x+ i) + φ(x− i) + φ(x+ j) + φ(x− j)]

}(2)

where x is a coordinate on a 2-dimensional grid withdimensions denoted by i and j and σ being a constant. Thisspecific choice of normalization allows repeatedly applying theoperator on a vector while keeping the numerical values of theelements to be of the same order. The neighboring elementsof a single element φ involved in the operation are shown inFigure 4. To calculate the element φ(x) on the left-hand-sideof Equation 2 we need to load the element φ(x) and the fourneighboring elements (φ(x+i), φ(x−i), φ(x+j) and φ(x−j))in the 2-dimensional grid presented in the Figure 4.

In Figure 3 we show four consecutive iterations of thestencil operation on an array A, assuming both input arrayA and output array B elements are stored in lexicographicsite order, i.e. the x-coordinate runs fastest, the y-coordinateslowest. Assuming A[y,x] is aligned in memory, then for anarchitecture with wide vector units, a number (which dependson the width of the unit) of following elements, A[y,x+1],A[y,x+2], etc., will not be aligned. Therefore, if one wereto vectorize this operation in order to make use of the vectorfloating point units, shuffle operations would be required toalign the appropriate elements at the right position of thevector register. For instance, A[y][x] is in the 0-th elementof the register once loaded, but needs to be in the 1-st elementof the register since it appears on the second line of Figure 3.

In Figure 3 we show with same color the same elements. Ascan be seen, a number of shuffle operations would be requiredto allow vectorization of this operation. In the followingsection, we will look into a data layout re-ordering which willreduce the number of shuffles required.

B. SU(3) Multiplication Kernel

The second kernel we implemented is the component ofthe Wilson Dirac equation which involves the multiplicationwith the gauge-links, which are elements of the special unitarygroup SU(3). As mentioned this is effectively a multiplicationof an array of complex 3×3 matrices u, with an array ofcomplex 3×4 ψ vectors (ψ fields are vectors in color space).

Complex number types, such as the C99 complex type,are usually stored as a structure or array of two elements,with the real part in the first element and the imaginary partin the second element. A complex multiplication thereforeinvolves cross-terms, i.e. multiplying the real part of the firstoperand with the imaginary part of the second element andvice-versa. If this operation is to be vectorized, one thereforeneeds to reshuffle the operands’ real and imaginary parts.

A more subtle point however is the fact that a gauge-linktakes up 144 bytes in double precision. If using registerswider than SSE, e.g. AVX (256-bit wide) or the MIC registers(512-bit wide), this means that not every gauge-link can bealigned in memory, unless consecutive gauge-links are padded.There are downsides in both solutions: padding simplifies thenumber of shuffle operations required because each gauge-link begins on an aligned memory boundary, but reduces theeffective bandwidth available. No padding complicates theshuffle operations required but makes better use of memorybandwidth. We will explore data layouts alternative to thesetwo in the next section.

IV. DATA LAYOUT OPTIMIZATIONS

In this section we describe four different techniques thatwe applied to the lattice data layout in order to create so-called “vector-friendly” layouts of the data, that allowed easiervectorization of the compute kernels. Our aim is to not restrictthese methods to the Intel Xeon Phi case of 512-bit registers,but to generalize our approach to vector units which aremultiple times the length of a double-precision variable. Someof the layouts we used were only applicable to the SU(3)multiplication kernel. For instance, we use the so-called Shortimplementation which is specific to SU(3) matrices. We alsouse Data Padding to observe the effect of padding betweengauge-links. Vector-friendly Ordering techniques were appliedto both kernels described in the previous section.

A. The Short Implementation

The fact that the gauge-links are elements of the SU(3)group has been exploited to reduce their memory requirements.Namely, by the definition of an SU(3) matrix, that its Hermi-tian conjugate is its own inverse: uu† = 1, one can reconstructthe ninth element from the other eight. The advantage to thisis that eight elements align in memory, and therefore everysuch truncated, or short, gauge-link will start on an alignedboundary. The disadvantage is that one has to recompute theninth element of the gauge-link every time it is to be used

B. Data Padding

Padding is used regularly when requiring that every elementof an array of structures be aligned in memory. We include aversion with padding of the gauge-links for completeness. Wenote however that an architecture with potentially very largevector units, of a size multiple times the size of a gauge link,would prohibit padding as an optimization technique.

C. The Vector-friendly Ordering Technique

Perhaps the simplest way of reordering data such as to alloweasier vectorization, is to change the order in which the indices

Fig. 5. The Hopping Vector-friendly Ordering technique.

run. In our case we have an array of gauge-links which arestored in memory with the color indices running fastest:

_Complex double u[NV][NC][NC];

where NV is the number of lattice sites and NC=3. Definingthe lattice sites to run faster however:

_Complex double u[NC][NC][NV];

would allow easier vectorization, with the restriction that thenumber of lattice sites NV is a multiple of the vector registerlength. With this data layout, the multiplication loop becomes:

for(c0=0; c0<NC; c0++)for(c1=0; c1<NC; c1++)

for(v=0; v<NV; v++) {phi[mu][c0][v] += u[c0][c1][v]*psi[mu][c1][v];

}

This layout is vector-friendly, because the v-loop can beeasily unrolled and vectorized. We call this the Vector-friendlyOrdering (VFO) technique. The disadvantage is that for largeNV this technique is not optimal in data reuse, since, forinstance, phi[mu][c0][v] must be retrieved three timesfrom memory.

D. The Hopping Vector-friendly Ordering Technique

In The Hopping Vector-friendly Ordering (Hopping VFO)follows naturally from the requirement for cache locality ofVFO above. The idea is to re-order the data such that part ofthe volume index v runs faster. The extent by which it runsfaster depends on the vector width. For instance one can definethe data arrays as:

_Complex double u[NV/NW][NC][NC][NW];

where NW is the number of _Complex double elementswhich fit in a vector register. E.g. for the case of the XeonPhi NW = 4, and for AVX NW = 2.

The implications of this layout are more pronounced whenone considers the 2D stencil operation. In this case, one wouldre-define the arrays from:

float phi[L][L];

to

float phi[L][L/NW][NW];

with the requirement that L is a multiple of NW. One thenre-orders the entries as such:

float phi[L][L];float pho[L][L/NW][NW];for(y=0; y<L; y++)for(x=0; x<L/NW; x++)

for(w=0; w<NW; w++)pho[y][x][w] = phi[L][w*L + x];

This allows vectorizing along the w component:

pho_out[y][x][w+0] = a*pho_in[y][x][w+0]+ b*(

pho_in[y+1][x][w+0] +pho_in[y-1][x][w+0] +pho_in[y+1][x][w+0] +pho_in[y-1][x][w+0]);

pho_out[y][x][w+1] = a*pho_in[y][x][w+1]+ b*(

pho_in[y+1][x][w+1] +pho_in[y-1][x][w+1] +pho_in[y+1][x][w+1] +pho_in[y-1][x][w+1]);

pho_out[y][x][w+2] = a*pho_in[y][x][w+2]+ b*(

pho_in[y+1][x][w+2] +pho_in[y-1][x][w+2] +pho_in[y+1][x][w+2] +pho_in[y-1][x][w+2]);

pho_out[y][x][w+3] = a*pho_in[y][x][w+3]+ b*(

pho_in[y+1][x][w+3] +pho_in[y-1][x][w+3] +pho_in[y+1][x][w+3] +pho_in[y-1][x][w+3]);

The only shuffle operations required are when [y][x] are ona boundary. In such a case one requires shuffle operations toimpose the periodic boundary condition, as shown in Figure 5.

V. EXPERIMENTAL EVALUATION

In this section, we describe our experimental setup andevaluate the performance of the proposed optimizations fromSection IV using the kernels described in Section III. We alsoevaluate the performance of two LQCD libraries, the QUDAparallel library for GPUs [14] and the QPhix library for theXeon processor family [15].

A. Experimental SetupWe used the 2D stencil operator and the SU(3) Multiplica-

tion kernel (described in Section III) to evaluate the perfor-mance of the Xeon Phi and the impact on the performance of

8 16 32 64 128 256 512 1024

0

20

40

60

80

100

120

140

1 4 8 16 30 60 120 240

Number of CUDA ThreadsB

and

wid

th (

GB

/s)

Number of Threads

Baseline

Auto-vectorization

VFO

NVIDIA K20m

Fig. 6. Bandwidth performance for the 2D stencil operation, for varyingnumber of threads

the optimizations we propose. The sources for the two kernelsare openly available on github [16]. We run our experimentsvarying the problem sizes for both kernels and for a varyingnumber of cores and a varying number of SMTs per core. Forup to 60 threads the number of threads per sore is 1, for 120threads it is 2 and for 240 threads it equals 4.

Our hardware setup was an Intel Xeon Phi 7120P with 61cores and 4 threads per core, totaling 244 threads. For betterperformance results we reserved one core for the operatingsystem and used the remaining 60 cores for the applicationexecution in Coprocessor-only mode. The coprocessor we usedhas a total of 16GB of main memory and runs at 1.238GHz. Tocross-compile our benchmarks for the Xeon Phi we used theIntel icc v.14.0.1 compiler with the -mmic flag indicating theMIC architecture and the -O3 optimization flag that includesauto-vectorization. For the GPU experiments we used anNVIDIA Kepler K20m card with 2496 CUDA cores. The totalmemory of the GPU card is 5GB and the working frequency is706GHz. To compile for the GPU we used the nvcc NVIDIAcompiler, v.5.5.0.

Finally, the performance metric we quote for the two kernelsis sustained bandwidth, that represents the number of bytesread or written during the total calculation, divided by theexecution time. The baseline shown in all the results of thetwo kernels is the original parallel implementation of thekernel with no vector optimizations. In all cases we repeatthe application several hundreds to several thousands of time,for a run-time of the order of one minute. The results presentedare averages over the number of repetitions. For the QPhix andQUDA libraries we present the performance as the number offloating-point operations per second.

B. Experimental Results

In this work we perform a scalability test of bandwidthperformance for two LQCD kernel applications on a singleXeon Phi using the Coprocessor-only mode of execution.We study the impact of vectorization, the limitations of thecompiler auto-vectorization when using complex number dataand solutions that allow the vectorization of complex data setsand increase the bandwidth performance. Apart from the twocompute kernels mentioned in Sections III-A and III-B we

0

20

40

60

80

100

120

140

1 4 8 16 30 60 120 240

Ban

dw

idth

(G

B/s

)

Number of Threads

Baseline

Auto-vectorization

Short

Padding

VFO

VFO - with manual vectorization

Hopping VFO with manual vectorization

Fig. 7. Bandwidth performance for the SU(3) Multiplication kernel forvarying number of threads and for the largest data set of our evaluation(3840K)

present performance results for two full LQCD applicationcodes, one for the Xeon Phi (QPhix) and one for NVIDIAGPUs (QUDA).

In Figure 6, we present the results of the 2D Stenciloperator. Compared to the baseline in our results, the auto-vectorization of the compiler manages around a 660% band-width increase. Our VFO technique seems to produce a slightoverhead when reordering the data that cannot be eliminatedfor up to 60 threads. When we use the SMT capabilities ofthe Xeon Phi and increase the number of threads per coreto 2 and 4, we manage to get as much performance as theauto-vectorization of the compiler. This is likely due to thehigher I/O throughput achieved by hiding the memory latencythrough the use of multiple threads.

For this kernel we also implemented a version for the GPU.The results show that for a small number of threads the GPUperformance is close to the Xeon Phi but when we increasethe number of threads on both coprocessors, the Xeon Phisignificantly outperforms the GPU. A possible explanation isthat the input size used (2048×2048 arrays size) for these testsis not large enough for the GPU to hide all memory latenciesand also not large enough to efficiently utilize all the CUDAthreads available.

The SU(3) Multiplication kernel uses complex floating pointarithmetic, which means that each element of the matrix hasa real and an imaginary part, with each one being a double-precision floating point number. In Figure 7, we present theperformance results of this kernel. The auto-vectorization ofthe compiler in this case seems to under-perform comparedto the baseline implementation. This is likely due to thedata dependencies involved in complex arithmetic, which mayhide auto-vectorization opportunities from the compiler. Thepotential performance offered by the vector capabilities ofthe Xeon Phi is therefore lost unless manual optimization isimplemented. To this end we evaluate the performance of aseries of data manipulation techniques and in all cases we keepthe vectorization flag of the compiler ON.

In the Short implementation, where we remove one element,we vectorize all operations including the calculation of themissing element. We observe that this change leads to a band-

0

50

100

150

200

15 60 240 960 3840

Ban

dw

idth

(G

B/s

)

Size of Input

Baseline Auto-vectorization

Short Padding

VFO VFO - with manual vectorization

Hopping VFO with manual vectorization

Fig. 8. Bandwidth performance for the SU(3) Multiplication kernel varyingthe input size and for 240 threads

width increases of ≈1.4x. Although this technique introducesadditional operations re-computing the missing element, theincreased performance attained by the vectorization seems tobe sufficient to hide this overhead. The same is observed forthe Padding implementation, where we pad the end of each umatrix in memory such that every matrix begins on an alignedmemory address. This method out-performs the bandwidth ofthe baseline scenario, by approximately the same factor as theshort implementation.

Applying the VFO ordering technique allows a morestraight forward vectorization of the kernels, with a signif-icantly smaller requirement on shuffle operations. As can beseen this technique introduces a significant penalty, performingmore than 3× worse than the baseline. We remind that whenreordering the complex data of the SU(3) kernel within thistechnique we inevitably reduce data re-use, and effectivelyincrease the number of cache misses, loosing both temporaland spatial locality of the data. This becomes more obviousin Figure 7, where we see the performance as a function ofthe problem size. We therefore conclude that the overheadimposed by the loss of locality in cache is greater thanthe gains achieved through use of the vector registers. Thisis expected in the cases of bandwidth-bound computationalkernels, such as this one.

We now turn to the Hopping VFO technique, which alignsthe data in vector registers in such a way that distant data arepacked within the same register. This data layout is intendedto on one hand ease the vecotrization of the codes by requiringless shuffles in the way VFO does, and on the other hand retaindata locality such that the excessive cache misses observed inVFO are mitigated. As can be seen, we observe an almost2-fold increase in the bandwidth compared to the baseline.In Figure 8, we also present results for different input sizesthat shows that the performance gain from the Hopping VFOtechnique is approximately constant over all input sizes.

In Figure 9, we show results obtained using QPhix, a fullLQCD implementation of the Wilson Dslash operator. Theseresults are intended to demonstrate the effect of combiningthese various vectorization techniques in a complete appli-cation. It is significant to notice that the results for QPhix

40 80 120 160 200 240

0

50

100

150

200

250

10 20 30 40 50 60

Number of Threads

Pe

rfo

rman

ce (

GFL

OP/

s)

Number of Cores

Dslash - SMT 1 Dslash - SMT 4

Fig. 9. QPhix scalability on Intel Xeon Phi with 1 and 4 threads per core

0

50

100

150

200

250

300

16x16x16 32x32x32 48x48x48P

erf

orm

ance

(G

FLO

P/s)

Input Size

Intel Xeon Phi Nvidia Tesla K20

Fig. 10. Sustained performance as a function of the problem size, for themaximum number of threads. Red circles for NVIDIA GPUs using the QUDAcode and blue crosses for the Intel Xeon Phi using the QPhix code.

show good performance scalability and also a large increasein performance when using the multi-threading capabilities ofthe Xeon Phi. This demonstrates that the Xeon Phi can indeedhide memory latencies by increasing I/O throughput via theuse of more threads per core, which utilize the shared, last-level cache. In Figure 10, we compare different input sizes forthe QPhix implementation on the Xeon Phi and the QUDAimplementation on the NVIDIA GPU. An average of 9% and7% of peak performance is achieved on the Xeon Phi and theGPU respectively.

VI. RELATED WORK

LQCD is an established HPC application consuming asignificant fraction of European and US supercomputing re-sources. LQCD researchers have already identified calcula-tions which will require Exa-scale computing. The LQCDcommunity is notorious for being an early adopter of newtechnologies, porting codes to new architectures that have thepotential of multiples increases in efficiency. Therefore LQCDcodes are usually the first to be ported to newly introducedHPC architectures.

There are various implementations of LQCD operators inthe literature that target different architectures. In this sec-tion we concentrate on those relevant to accelerator-basedarchitectures. M.A. Clark et al. in [17], [18], present theQUDA library which was used in this work, for performing

calculations in lattice QCD on GPUs, leveraging NVIDIA’sCUDA [19] programming platform. In a continuation of thiswork, Ronald Babich et al. present the parallel implementationof the QUDA library for multi-GPU calculations [14], thatmanages to increase the performance of both single- anddouble-precision calculations on clusters with multiple GPUs.

Simon Heybrock et al. in [20], propose a domain decompo-sition for the LQCD Wilson-Clover operator with single- andhalf-precision operations on the Intel Xeon Phi coprocessor.Using domain-decomposition methods they manage to reducethe data movement from and to main memory as well as viathe network. In [21], the authors present their approach ofimplementing the Wilson Dslash operator for the Intel XeonPhi coprocessor while using cache-blocking techniques andblock-to-core methods for increasing the performance, whilean evolution of this work for multi-node Xeon Phi clusters ispresented in [22]. Their resulting Qphix implementation wasused in some of our experiments, in which we can reproducetheir performance results.

VII. CONCLUSIONS

In this work we implemented two kernels that derive fromthe computational-intensive simulations of LQCD, namelyfrom the Wilson Dslash operator. We implemented a 2D stenciloperator that operates on single-precision floating point dataand the SU(3) Multiplication kernel that operates on complex(double-precision) data. Using these kernels we study waysof exploiting the parallelism offered by the wide vector unitsof the Intel Xeon Phi coprocessor. Our results show that theperformance of LQCD-based applications can significantlyincrease if moderate effort is devoted in optimizing the datalayout. In the process we found that the compiler auto-vectorization was limited to non complex arithmetic, andwas unable to automatically vectorize the code in complexarithmetic kernels. We evaluated various methods for manip-ulating the data to solve this problem and we found thatthe Hopping Vector-friendly Ordering technique allows oneto efficiently vectorize the application. Using this techniquewe can achieve ≈1.8x performance increase for the SU(3)Multiplication kernel that uses complex data structures.

In the future, we plan to test our implementation on a largercluster of multiple nodes equipped with multiple Xeon Phicoprocessors. We will study the impact of an interconnectnetwork on the applications and the penalty this might haveon performance, in conjunction with the data manipulationtechniques used in this work.

ACKNOWLEDGMENT

A. Diavastos and G. Stylianou are supported by theCyprus Research Promotion foundation, under project “GPUClusterware”(TΠE/ΠΛHPO/ 0311(BIE)/09).

REFERENCES

[1] (2015, Feb.) Top500.org. [Online]. Available: http://top500.org/[2] Titan Supercomputer. [Online]. Available:

https://www.olcf.ornl.gov/titan/

[3] Piz Daint Swiss Supercomputer. [Online]. Available:http://www.cscs.ch/computers/piz daint

[4] Cray Cs-Storm Accelerator-optimized Cluster Supercomputer. [Online].Available: http://www.cray.com/products/computing/cs-series/cs-storm

[5] Tianhe 2 supercomputer. [Online]. Available:http://en.wikipedia.org/wiki/Tianhe-2

[6] Stampede: Texas Advanced Computing Center. [Online]. Available:https://www.tacc.utexas.edu/stampede/

[7] P. Boyle, C. Jung, and T. Wettig, “The QCDOC supercomputer: Hard-ware, software, and performance,” eConf, vol. C0303241, p. THIT003,2003.

[8] F. Bodin, P. Boucaud, N. Cabibbo, F. Calvayrac, M. Della Morte et al.,“The apeNEXT project,” Nucl.Phys.Proc.Suppl., vol. 106, pp. 173–176,2002.

[9] G. I. Egri, Z. Fodor, C. Hoelbling, S. D. Katz, D. Nogradi et al., “LatticeQCD as a video game,” Comput.Phys.Commun., vol. 177, pp. 631–639,2007.

[10] The Intel Xeon Phi coprocessor. [Online]. Avail-able: http://www.intel.com/content/www/us/en/processors/xeon/xeon-phi-detail.html

[11] R. Rahman, Intel R© Xeon PhiTM

Coprocessor Architecture and Tools:The Guide for Application Developers. Apress, 2013.

[12] K. G. Wilson, “Confinement of quarks,” Physical Review D, vol. 10,no. 8, p. 2445, 1974.

[13] A. Kowalski and X. Shen, “Implementing the dslash operator in opencl,”College of William and Mary Technical Report, 2010.

[14] R. Babich, M. A. Clark, and B. Joo, “Parallelizing the QUDA library formulti-GPU calculations in lattice quantum chromodynamics,” in HighPerformance Computing, Networking, Storage and Analysis (SC), 2010International Conference for. IEEE, 2010, pp. 1–11.

[15] JeffersonLab. QPhiX: QCD for Intel Xeon Phi and Xeon Processors.[Online]. Available: https://github.com/JeffersonLab/qphix

[16] G. Koutsou and G. Stylianou. (2014, Apr.) GPU Clusterware. [Online].Available: https://github.com/gpucw

[17] M. A. Clark, R. Babich, K. Barros, R. C. Brower, and C. Rebbi, “Solvinglattice qcd systems of equations using mixed precision solvers on gpus,”Computer Physics Communications, vol. 181, no. 9, pp. 1517–1528,2010.

[18] QUDA: A library for QCD on GPUs. [Online]. Available:http://lattice.github.io/quda/

[19] “NVIDIA, CUDA, programming guide,” 2008.[20] S. Heybrock, B. Joo, D. D. Kalamkar, M. Smelyanskiy, K. Vaidyanathan,

T. Wettig, and P. Dubey, “Lattice qcd with domain decompositionon intel R© xeon phi co-processors,” in High Performance Computing,Networking, Storage and Analysis, SC14: International Conference for.IEEE, 2014, pp. 69–80.

[21] B. Joo, D. D. Kalamkar, K. Vaidyanathan, M. Smelyanskiy, K. Pamnany,V. W. Lee, P. Dubey, and W. Watson III, “Lattice qcd on intel R© xeonphitm coprocessors,” in Supercomputing. Springer, 2013, pp. 40–54.

[22] K. Vaidyanathan, K. Pamnany, D. D. Kalamkar, A. Heinecke,M. Smelyanskiy, J. Park, D. Kim, A. Shet, B. Kaul, B. Joo et al., “Im-proving communication performance and scalability of native applica-tions on intel xeon phi coprocessor clusters,” in Parallel and DistributedProcessing Symposium, 2014 IEEE 28th International. IEEE, 2014, pp.1083–1092.