Spark-MPI: Approaching the Fifth Paradigm ofCognitive Applications

Nikolay MalitskyBrookhaven National Laboratory

Ralph CastainIntel Corporation

Matt CowanBrookhaven National Laboratory

Abstract—Over the past decade, the fourth paradigm of data-intensive science rapidly became a major driving concept ofmultiple application domains encompassing and generating large-scale devices such as light sources and cutting edge telescopes.The success of data-intensive projects subsequently triggeredthe next generation of machine learning approaches. These newartificial intelligent systems clearly represent a paradigm shiftfrom data processing pipelines towards the fifth paradigm ofcomposite cognitive applications requiring the integration of BigData processing platforms and HPC technologies. The paper ad-dresses the existing impedance mismatch between data-intensiveand compute-intensive ecosystems by presenting the Spark-MPIapproach based on the MPI Exascale Process ManagementInterface (PMIx). The approach is demonstrated within thecontext of hybrid MPI/GPU ptychographic image reconstructionpipelines and distributed deep learning applications.

I. INTRODUCTION

The fourth paradigm of data-intensive science, coined byJim Gray [1], rapidly became a major conceptual approachfor multiple application domains encompassing and generatinglarge-scale scientific drivers such as fusion reactors and lightsource facilities [2] [3]. Taking its root from data managementtechnologies, the paradigm emphasized and generalized a data-driven knowledge discovery direction that complemented thecomputational branch of scientific disciplines. The success ofdata-intensive projects subsequently triggered an explosion ofnumerous machine learning approaches [4] [5] [6] addressinga wide range of industrial and scientific applications, suchas computer vision, self-driving cars, and brain modelling,just to name a few. The next generation of artificial intelli-gent systems clearly represents a paradigm shift from dataprocessing pipelines towards knowledge-centric applications.As shown in Fig. 1, these systems broke the boundariesof computational and data-intensive paradigms and began toform a new ecosystem by merging and extending existingtechnologies. Identifying this trend as the fifth paradigm aimsto infer common aspects among diverse cognitive computingapplications and steer the development of complementarysolutions for addressing emerging and future challenges.

The initial landscape of data-intensive technologies wasdesigned after Google’s Big Data stack over 15 years ago.It represented a consolidated scalable platform bringing to-gether database and computational technologies. The open-source version of this platform was further advanced withthe Spark framework resolving the immediate requirementsof numerous data-intensive projects. The new model of the

Spark computing platform significantly extended the scopeof data-intensive applications, spreading from SQL queriesto machine learning to graph processing. According to thedata-information-knowledge-wisdom model [7], these projectseventually elevated data-information pipelines to practical ap-plications of knowledge development. Cognitive systems ofthe fifth paradigm take the relay baton from data-driven pro-cessing pipelines and generalizes their scope with knowledgeacquisition processes carried out by rational agents throughthe exploration of their environments.

Fig. 1. The Fifth Paradigm. The diagram shows the conceptual structure ofthe new paradigm integrating resources from both its ancestors, computationaland data-intensive sciences, for building cognitive computing applications.

In contrast with the MapReduce embarrassingly parallelpipelines, machine learning applications rely on the commu-nication among distributed workers for synchronizing theirinternal representations. This mismatch led to the developmentof new distributed processing frameworks, such as GraphLab[8], CNTK [9], TensorFlow [10], and Gorila [11]. As with anystandard evolutionary spiral, a variety and growing number ofdifferent approaches eventually raised the question of theirconsolidation. Similar problems are faced by researchers oflarge-scale scientific experimental facilities and computationalprojects. Prior to the Big Data era, most scientific algorithmswere built within the third computational paradigm based onHPC clusters and Message Passing Interface (MPI) commu-nication model. To address the immediate requirements ofemerging applications, the fourth data-intensive paradigm wasdeveloped by minimally intersecting with the HPC ecosystemas shown in Fig. 1.

arX

iv:1

806.

0111

0v1

[cs

.DC

] 1

6 M

ay 2

018

The strategic transition from data-intensive science towardsthe fifth paradigm of composite cognitive computing applica-tions is a long-term journey with many unknowns. This paperaddresses the existing mismatch between Big Data and HPCapplications by presenting the Spark-MPI integrated platformaiming to bring together Big Data analytics, HPC scientificalgorithms and deep learning approaches for tackling newfrontiers of data-driven discovery applications. The remainderof the paper is organized as follows. Section 2 provides abrief overview of the Spark data-intensive and MPI high-performance platforms and outlines the Spark-MPI integratedapproach based on the MPI Process Management Interface(PMI). Section 3 and Section 4 further elaborate the approachwithin the context of the hybrid MPI/GPU ptychographicimage reconstruction pipelines and distributed deep learningapplications. Section 5 provides insights into future directionsusing the PMI-Exascale library. Finally, Section 6 and Section7 survey related work and conclude with a summary.

II. SPARK-MPI PLATFORM

The integration of data-intensive and compute-intensiveecosystems was addressed by several projects. For exam-ple, Geoffrey Fox and colleagues provided a comprehensiveoverview of the Big Data and HPC domains. Their applicationanalysis [12] was based on several surveys, such as the NISTBig Data Public Working Group and NRC reports, includingmultiple application areas: energy, astronomy and physics,climate, and others. Overall, from a conceptual perspective,the Spark-MPI platform can be considered as the Spark-basedversion of the Exascale Fast Forward (EFF) I/O Stack three-tier architecture [13]. The Spark-MPI platform furthermorefocuses on the development of a common integrated approachaddressing a wide spectrum of applications including large-scale computational studies, data-information-knowledge dis-covery pipelines for experimental facilities, and reinforcementlearning systems.

Fig. 2 shows a general overview of the Spark-MPI integratedenvironment. It is based on the Spark Resilient DistributedDataset (RDD) middleware [14] that decouples various datasources from high-level processing algorithms. RDDs aredistributed fault-tolerant collections of in-memory objects thatcan be processed in parallel using a rich set of operations,transformations and actions. The top layer is represented by anopen collection of high-level components addressing differenttypes of data models and processing algorithms, including ma-chine learning and graph processing. The interfaces betweenRDDs and distributed data sources are provided by Connectorsthat are already implemented for major databases and filesystems.

In addition, Spark is designed to cover a wide range ofworkloads that previously required separate distributed sys-tems encompassing batch applications, iterative algorithms,interactive queries, and streaming. This combination formsa powerful processing ecosystem for building data analysispipelines and supporting multiple higher-level components

specialized for various workloads. The Spark Streaming mod-ule [15] further extends the RDD pluggable mechanism withthe Receiver framework enabling to ingest real-time data intoRDDs from streaming sources, such as Kafka and ZeroMQ.Adherence to the RDD model automatically provided theSpark streaming applications with the same functional inter-face and strong fault-tolerance guarantees including exactly-once semantics.

The combination of a data-intensive processing frameworkwith a consolidated collection of diverse data analysis al-gorithms offered by Spark represents a strong asset for itsapplication in large-scale scientific projects across differentphases of the data-information-knowledge discovery path. Incontrast with existing data management and analytics systems,the Spark in-situ approach does not require the transformationof data into different formats and provides a generic interfacebetween heterogeneous algorithms with heterogeneous datasources. The current version of the Spark programming model,however, is limited by the embarrassingly parallel paradigmand the Spark-MPI approach serves to extend the Sparkecosystem with the MPI-based high-performance computa-tional applications.

MPI is an abbreviation for the Message Passing Interfacestandard that is developed and maintained by the MPI Forum[16]. The process of creating the MPI standard began in April1992 and as of now, it is used in most HPC applications.The popularity of the MPI standard was determined by theoptimal combination of concepts and methods challenged bytwo conflicting requirements: scope of parallel applicationsand portability across different underlying communicationprotocols.

The MPI standard interface extends the Spark embarrass-ingly parallel model with a rich collection of communica-tion methods encompassing Remote Memory Access (RMA),pairwise point-to-point operations (e.g., send and receive),master-worker (e.g., scatter and gather) and peer-to-peer (e.g.,allreduce) collective methods. In addition, the Barrier methodwithin the collective category provides a synchronizationmechanism for supporting the Bulk Synchronous Parallelparadigm. To address the scalability and performance aspects,MPI introduced the concept of Communicators that defined thescope for communication operations. As a result, this approachsignificantly facilitated the development and integration ofparallel libraries using inter- and intra-communicators.

To support the MPI parallel model across different operatingand hardware systems, the MPI frameworks are based on aportable access layer. One of its initial specifications, AbstractDevice Interface (ADI [17]), was developed within the MPICHproject. Later, the MVAPICH project further extended theADI implementations to support InfiniBand interconnects andGPUDirect RDMA [18]. The OpenMPI team introduced adifferent solution, Modular Component Architecture (MCA)[19], that was derived as a generalization of four projects [20]bringing together over 40 frameworks. MCA utilizes compo-nents (a.k.a. plugins) to provide alternative implementationsof key functional blocks such as message transport, mapping

Fig. 2. Conceptual architecture overview of the Spark-MPI platform

algorithms, and collective operations. As a result, OpenMPIByte Transfer Layer (BTL) represents an open collection ofnetwork-specific components for supporting shared memory,TCP/IP, OpenFrabric verbs, and CUDA IPC, just to name afew.

Running parallel programs on HPC clusters requires inter-actions with external process and resource managers, suchas SLURM and Torque, to enable the MPI processes todiscover each other’s communication endpoints. Within theMPI ecosystem, this topic is typically addressed by the ProcessManager Interface (PMI [21]). While the implementation ofthe PMI specification was never standardized, libraries nearlyalways consist of two parts: client and server. The clientcode is linked with the MPI program and provides messagingsupport to the server - it has no a priori knowledge of theoverall application, and must rely on the server to provide anyrequired information.

The PMI server is instantiated on each node that supports anMPI process and has both the ability to communicate with itspeers (usually over an out-of-band Ethernet connection) andfull knowledge of how to contact those peers (e.g., the socketupon which each peer is listening). The server is typicallyeither embedded in the local daemon of the system’s resourcemanager, or executed as a standalone daemon started by acorresponding launcher such as mpiexec.

The Spark-MPI approach extends the scope of the PMImechanism for integrating the Spark and MPI frameworks.Specifically, it complemented the Spark conventional driver-worker model with the PMI server-worker interface for estab-lishing MPI inter-worker communications as outlined in Fig.3 and Fig. 4.

The first version of the Spark-MPI approach was validatedwith the primary internal process manager, Hydra, used by two

Fig. 3. Spark-MPI approach

Fig. 4. Sequence diagram of the Spark-MPI approach

MPI projects (MPICH and MVAPICH). The Hydra ProcessManager (PM) is started by the MPI launcher mpirun onthe launch node which subsequently spawns a tree-basedcollection of intereconnected proxies on the allocated nodes.Each proxy locally spawns one or more application processesand then acts as the PMI server for those processes. Duringinitialization, each process “publishes” its connection infor-mation to the proxy, which then performs a global collectiveoperation to share the information across all proxies for

eventual distribution to the application. Within the Spark-MPIintegrated platform, MPI application processes are started bythe Spark scheduler (see Fig. 4). Hydra local proxies thereforewere modified to suppress their launching functionality.

Recently, the Spark-MPI approach was integrated with theOpen MPI framework. The OpenMPI Modular ComponentArchitecture further streamlined its implementation as thesparkmpi plugin of the OpenRTE Daemon’s Local LaunchSubsystem (ODLS). The following sections will demonstratethis approach within the context of the hybrid MPI/GPU pty-chographic image reconstruction pipelines and deep learningapplications.

III. PTYCHOGRAPHIC IMAGE RECONSTRUCTIONPIPELINES

Ptychography is one of the essential image reconstructiontechniques used in light source facilities. It was originallyproposed for electron microscopy [22] and lately applied toX-ray imaging [23] [24]. The method consists of measuringmultiple diffraction patterns by scanning a finite illumination(also called the probe) on an extended specimen (the object).The redundant information encoded in overlapping illuminatedregions is then used for reconstructing the sample transmissionfunction. Specifically, under the Born and paraxial approxima-tions, the measured diffraction pattern for the jth scan positioncan be expressed as:

Ij(q) =| Fψj |2 (1)

where F denotes Fourier transformation, q is a reciprocalspace coordinate, and ψj represents the wave at the exit ofthe object O illuminated by the probe P:

ψj = P(r− rj)O(r) (2)

Then, the object and probe functions can be computed fromthe minimization of the distance ‖Ψ−Ψ0‖2 as [25]:

ε = ‖Ψ−Ψ0‖2 = ΣjΣr | ψj(r)− P0(r− rj)O0r |2 (3)

∂ε

∂P0= 0 : P0(r) =

Σjψj(r + rj)O∗(r + rj)

Σj | O(r + rj) |2(4)

∂ε

∂O0= 0 : O0(r) =

Σjψj(r)P∗(r + rj)

Σj | P(r− rj) |2(5)

These minimization conditions need to be augmented withthe modulus constraint (1) and included in the iteration loop.For example, the comprehensive overview of different iterativealgorithms is provided by Klaus Giewekemeyer [26]. At thistime, the difference map [27] is considered as one of themost generic and efficient approaches to address these types ofimaging problems. It finds a solution in the intersection of twoconstraint sets using the difference of corresponding projectionoperators, π1 and π2, composed with associated maps, f1 andf2:

ψn+1 = ψn + β∆(ψn)

∆ = π1 ◦ f2 − π2 ◦ f1 (6)fi(ψ) = (1 + γi)πi(ψ)− γiψ

where γ1,2 are relaxation parameters. In the context of ptycho-graphic applications, these projection operators are associatedwith the modulus (1) and overlap (2) constraints. By selectingdifferent values of relaxation parameters, the difference map(6) can be specialized to different variants of phase retrievalmethods and hybrid projection-reflection (HPR) algorithms.Further developing HPR, Russel Luke [28] introduced therelaxed averaged alternating reflections (RAAR) approach:

ψn+1 = [2βπ0πa + (1− 2β)πa + β(1− π0)]ψn (7)

The RAAR algorithm was implemented in the SHARP pro-gram [31] at the Berkley Center for Advanced Mathematicsfor Research Applications (CAMERA).

SHARP is a high-performance distributed ptychographicsolver using GPU kernels and the MPI protocol. Since mostequations with the exception of (4) and (5) are framewiseintrinsically independent, the ptychographic application is nat-urally parallelized by dividing a set of data frames amongmultiple GPUs. Then, for updating a probe and an object,the partial summations of (4) and (5) are combined acrossdistributed nodes with the MPI Allreduce method as shown inFig. 5.

Fig. 5. MPI communication model of the SHARP solver

The SHARP multi-GPU approach significantly boosted theperformance of ptychographic applications at the NSLS-IIlight source facility and immediately highlighted the pathfor developing near-real-time processing pipelines. Table Icompares the performance results for processing 512 frameson different numbers of GPUs.

TABLE IBENCHMARK RESULTS OF THE SHARP-NSLS2 APPLICATION

Application Time (s) vs Number of GPUs (TESLA K80)1 2 4

SHARP-NSLS2 22.7 13.6 8.6

In the experimental settings, the time interval betweenframes takes approximately 50 ms, in other words 25 sec-onds for 512 frames. And according to Table I, the Spark-MPI application demonstrated the feasibility of the near-real-time scenario. This direction is especially important from theperspective of a new category of emerging four-dimensionaltomographic applications that combine series of ptychographicprojections generated at different angles of object rotation.

In these experiments, each ptychographic projection is recon-structed from tens of thousands of detector frames and theMPI multi-GPU version becomes critical for addressing theGPU memory challenges.

The Spark-MPI integrated platform immediately provideda connection between MPI applications and different typesof distributed data sources including major databases and filesystems. Furthermore, the Spark Streaming module reused andextended the RDD-based batch processing framework witha new programming abstraction called discretized stream, asequence of RDDs, processed by micro-batch jobs. Thesenew batches are created at regular time intervals. Similar tobatch applications, streams can be ingested from multiple datasources like Kafka, Flume, Kinesis and TCP sockets.

For evaluating the Spark-MPI approach, the SHARP ptycho-graphic pipeline was tested with the Kafka streaming platform,an Apache open source project that was originally developedat LinkedIn [30]. The corresponding simulation-based scenariois described with a conceptual diagram (see Fig. 6).

Fig. 6. Streaming demo with the Spark-MPI approach

According to this scenario, the input stream represents asequence of micro-batches. The Spark driver waits for a topic-init record and processes each micro-batch with the run batchmethod. At the beginning of this method, the Kafka dataare ingested into the Spark platform as the Kafka RDDs. Toachieve a higher level of parallelism, records of micro-batchesare divided into topics that are consumed by Kafka Receiverson distributed Spark workers. Each Kafka Receiver creates atopic-specific RDD and the Spark driver logically combinesthem together with a union operation. As a result, it preparesa distributed RDD to be processed with the MPI application.

The acceleration of image processing algorithms with thenext generation of GPU devices further strengthened thedirection by creating the necessary conditions for augment-ing photographic pipelines with optimization procedures. Themodern ptychographic approaches depend on many param-eters and their choice is important for achieving the mostaccurate reconstruction results. For example, Fig. 7 and Fig. 8demonstrate reconstructed object phases for different choicesof constraints.

Finding the most optimal parameters can be automatedwith conventional optimization approaches. In addition, thepipelines can be further advanced with modern machine learn-

Fig. 7. Object phases reconstructed from 40,000 frames (object constraints:amp max = 1.0, amp min = 0.0, phase max = π/2, phase min = - π/2)

Fig. 8. Object phases reconstructed from 40,000 frames (object constraints:amp max = 1.0, amp min = 0.95, phase max = 0.01, phase min = - 0.1)

ing techniques for image analysis and steering reconstructionalgorithms.

IV. DEEP LEARNING APPLICATIONS

The Spark-MPI platform was designed for building a newgeneration of composite data-information-knowledge discov-ery pipelines for experimental facilities. Deep learning appli-cations advanced the scope and requirements of large-scalescientific projects to the next level. From the perspectiveof the fifth paradigm, Spark-MPI can be considered as ageneric front-end of composite agent models for interactingwith heterogeneous environments.

Historically, distributed deep learning frameworks weredeveloped within the fourth paradigm of data-intensive pro-cessing platforms. On the other hand, compute-intensive taskshave been already successfully addressed with a HPC stackof hardware and MPI applications from the third computa-tional paradigm. The parallel acceleration of deep learningalgorithms was then pursued by several MPI-based projects,such as CNTK [9], TensorFlow-MaTex [31], FireCaffe [32],and S-Caffe [33].

CNTK and TensorFlow are deep learning toolkits developedby Microsoft and Google, respectively. For distributed training,CNTK relies on the MPI communication platform and can bedirectly deployed on HPC clusters. In contrast, the originalimplementation of the TensorFlow distributed version is basedon Google’s gRPC interface developed for cloud computingsystems using Ethernet. To leverage the HPC low latencyinterconnects, the TensorFlow-MaTEx project added two new

TensorFlow operators, Global Broadcast and MPI Allreduce,and correspondingly modified the TensorFlow runtime. TheFireCaffe and S-Caffe distributed approaches were developedaround single-node Caffe deep learning solvers according tothe data-parallel architecture. In addition, they further accel-erated the data-parallel communication schema by replacinga parameter server with the allreduce communication patternbased on a reduction tree. Recently, the TensorFlow projectwas extended with a hybrid communication interface basedon the combination of gRPC and MPI protocols. In contrastwith other deep learning applications, the TensorFlow frame-work provides a pluggable mechanism for registering differentcommunication interfaces that can be interchanged with othermore advanced or application-specific versions.

Within the beamline composite pipeline platform, the Spark-MPI approach was evaluated with Horovod [34], a MPI train-ing framework for TensorFlow. The Horovod team adoptedBaidu’s approach [35] based on the ring-allreduce algo-rithm [36] and further developed its implementation with theNVIDIA’s NCCL library for collective communication. As aresult, the ring-allreduce approach replaced parameter serversof the TensorFlow distributed version with an efficient mecha-nism for averaging gradients among the TesnorFlow workers.Their integration with the Horovod distributed framework con-sists of two primary steps as illustrated by the horovod trainmethod in Fig. 9. First, Horovod and MPI is initialized withhvd.init. And then, the TensorFlow worker’s optimizer iswrapped by hvd.DistributedOptimizer, a Horovod’s ring-allreduce distributed adapter.

Fig. 9. Horovod-TensorFlow example

The Spark-MPI pipelines enable to process the same methodon Spark Workers within Map operations as shown in Fig. 10.To establish MPI communication among the Spark Workers,the Map operation needs only to define PMI-related environ-mental variables (such as PMIX RANK and a port number)for connecting the Horovod MPI application with the PMIserver.

Implementing deep learning applications on the MPI paral-lel framework immediately extended the scope of the Spark-MPI ecosystem with composite pipelines as shown in Fig. 11.

For light source facilities, the development of compositepipelines involves two major topics: application of deep learn-ing approaches for analyzing reconstructed images and de-

Fig. 10. Spark-MPI-Horovod example

Fig. 11. End-to-end machine learning pipeline including reconstruction, imageanalysis, and feedback loop

velopment of machine learning feedback systems for steeringreconstruction algorithms. According to the survey by GeertLitjens and colleagues [37], deep learning techniques pervadeevery aspect of medical image analysis: detection, segmen-tation, quantification, registration, and image enhancement.The feedback system can be viewed from the perspective ofa rational agent that interacts with a reconstruction pipelinerepresenting its environment. Depending on the applications,an agent can be built with different learning techniques. Oneof the most important breakthroughs is associated with theintroduction of a deep Q-network (DQN) model for reinforce-ment learning [5]. The DQN-based approach demonstratedstate-of-the-art results in various applications [38] rangingfrom playing video games to robotics. As shown in Fig. 11,Spark-MPI provides a generic front-end for distributed deepreinforcement learning platforms on the HPC cluster.

V. RELATED WORK

The deployment of the Spark platform on HPC clustersand its comparison with the MPI approaches has been ad-dressed by several projects. The Ohio State University team[39] proposed an RDMA-based design for the data shuffleof Spark over InfiniBand. Alex Gittens and colleagues [40]demonstrated the performance gap between a close-to-metalparallelized C version and the Spark-based implementation ofmatrix factorization. To resolve this gap, they introduced theAlchemist system for socket-based interfacing between Sparkand existing MPI libraries. Michael Anderson and colleagues[41] proposed an alternative approach based on the Linuxshared memory file system. The third solution suggested by

Cyprien Noel, Jun Shi and Andy Feng from the Yahoo Big MLteam extended the Spark embarrassingly parallel model withthe RDMA inter-worker communication interface. Later, thisapproach was reused by the Sharp-Spark project [42] withinthe context of ptychographic reconstruction applications. TheSharp-Spark approach followed the Yahoo Big ML peer-to-peer model and augmented it with a RDMA address exchangeserver that significantly facilitated the initialization phaseresponsible for establishing Spark inter-worker connections.As a result, the RDMA address exchange server captured thePMI functionality of the MPI implementations and provideda natural transition to the PMI-based Spark-MPI approach.

The similarity between the Spark driver-worker computa-tional model and the data-parallel approach of deep learn-ing solvers triggered the development of a new categoryof applications such as SparkNet [43], CaffeOnSpark [44],TensorFlowOnSpark [45], and BigDL [46]. SparkNet directlyrelied on the Spark driver-executor scheme consisting of asingle driver and multiple executors running the Caffe orTensorFlow deep learning solvers on its own subset of data.In this approach, a driver communicates with executors foraggregating gradients of model parameters and broadcastingaveraged weights back for subsequent iterations. Accordingto the SparkNet-based benchmark, the driver-executor schemehowever introduced a substantial communication overhead thatwas minimized by subdividing the optimization loop intochunks of iterations. Addressing the same problem, the Caf-feOnSpark team proposed extending the Spark model with aninter-worker interface providing a MPI Allreduce style methodover Ethernet or InfiniBand. Later, the same team began theTensorFlowOnSpark project based on their RDMA extensionto the TensorFlow distributed platform. In comparison withthese projects, Spark-MPI aims to derive an application-neutralmechanism based on the MPI Process Management Interfacefor the effortless integration of Big Data and HPC ecosystems.

VI. PATH TOWARDS EXASCALE APPLICATIONS

The validation of the Spark-MPI conceptual solution es-tablished a basis for advancing this approach towards theproduction programming model based on the PMI-Exascale(PMIx) framework. Furthermore, this direction aligns withproposed changes to the MPI standard [47] being supportedby the PMIx community.

PMIx [48] was created in response to the ever-increasingscale of supercomputing clusters, and the emergence of newprogramming models such as Spark that rely on dynamicallysteered workflows. The PMIx community has therefore fo-cused on extending the earlier PMI work, adding flexibilityto existing APIs (e.g., to support asynchronous operations) aswell as new APIs that broaden the range of interactions withthe resident resource manager.

The initial version of the PMIx standard focused on resolv-ing the scaling challenges faced by bulk-synchronous program-ming models operating in exascale systems [49] [50]. How-ever, version 2 of the standard directly addressed the needsof dynamic, asynchronous programming models by providing

APIs for changing resource allocations (both adding andreturning resources, including the ability to “lend” resourcesback to the resource manager for limited periods of time);controlling application execution (e.g., ordering terminationand/or migration of processes, and coordinating requests forapplication preemption); notification of events such as connec-tion requests and process failures; and connections to serversfrom “unknown” processes not started by the server.

The Spark-MPI programming model utilizes the last featureas a mechanism by which the processes started by the Sparkscheduler can connect to a local PMIx server. The PMIx libraryincludes methods for automatically authenticating connectionsto the server based on a plugin architecture, thus allowing forready addition of new methods as required. Servers store theirrendezvous information in files located under system-definedlocations for easy discovery, and the client library executes asearch algorithm to automatically find and connect to a serverduring initialization.

Once connected to the server, the PMI-aware processescan utilize PMIx to asynchronously request connections toone or more processes. The connect and disconnect APIsin version 2 of the PMIx Standard retain support for bulk-synchronous programming models such as today’s MPI whileproviding the extensions needed for asynchronous models.Both require that the operation be executed as a collective,with all specified processes participating in the operation priorto it being declared complete - i.e., all processes specifiedin a call to PMIx Connect must call that API in order tocomplete the operation. In addition, the standard requires thatthe host resource manager (RM) treat the specified processesas a new “group” when considering notifications, termination,and other operations, and that no request to “disconnect” froma connected group be allowed to complete until all collectivesinvolving that group have also completed.

Finally, the PMIx community recognized that program-ming libraries have continued to evolve towards more of anasynchronous model where processes regularly aggregate intogroups that subsequently dissolve after completing some setof operations. These new approaches would benefit from anability to notify other processes of a desire to aggregate,and to allow the aggregation process itself to take placeasynchronously.

Accordingly, calls by PMI-aware processes toPMIx Connect are first checked by the PMIx server tosee if the other specified participants are available and havealso called PMIx Connect - if so, then the connection requestwill result in each involved process receiving full informationabout the other participating processes (location, endpointinformation, etc.) plus a callback containing the namespaceassigned to the connected group. The latter can be consideredthe equivalent of a communicator and used for constructingthat object.

If one or more of the indicated processes has not executedits call to PMIx Connect, then the server will issue an eventnotification requesting that the process do so. Applicationprocesses can register callback functions to be executed upon

receipt of a corresponding event, and events are cached sothey can be delivered upon process startup if the event isgenerated before that occurs. Once receiving a connectionrequest event, the process is given sufficient information in thenotification to allow it to join the requesting group, therebycompleting the collective operation. Applications can providean optional timeout attribute to the call to PMIx Connect sothe operation will terminate if all identified participants fail torespond within the given time limit.

Note that any single process can be simultaneously engagedin multiple connect operations. For scalability, PMIx does notuse a collective to assign a global identifier to the connectoperation, instead utilizing the provided array of process IDsas a default method to identify a specific PMIx Connect oper-ation. Applications can extend the ability to execute multipleparallel operations by providing their own string identifier foreach collective as an attribute to the PMIx Connect API. Notethat all partitipants in a given collective are required to callPMIx Connect with the same attribute value.

In cases where the involved hosts are controlled by differentRMs, the namespace identifier provided by the host RM for usein PMIx is no longer guaranteed to be unique, thereby leadingto potential confusion in the process identifiers. Accordingly,PMIx defines a method for resolving any potential namespaceoverlap by modifying the namespace value for a given processidentifier to include a clusteridentifier - a string name forthe cluster that is provided by the host RM, or applicationitself in the case of non-managed hosts.

The accumulated features of the PMIx distributed frame-work are identified as a new Exascale cluster service that sup-plements the conventional resource management and schedul-ing platform for gluing together HPC and Big Data ap-plications. On HPC clusters, support for PMIx is currentlyintegrated with the Open MPI run-time environment andSimple Linux Utility for Resource Management (SLURM[51]). Therefore, the deployment of the Spark-MPI platformon HPC clusters will be streamlined by adding SLURMinto the list of Spark schedulers (see Fig. 2): Standalone,YARN [52], Apache Mesos [53] and Kubernetes [54]. Asillustrated by the Spark-MPI ptychographic and deep learningexamples, this deployment approach is consistent with theSpark computational model. Furthermore, the asynchronousmodels supported by the PMIx framework highlight the nextdirection for deploying reinforcement learning architectures[55] on HPC clusters.

VII. CONCLUSIONS

The paper addresses the existing mismatch between BigData and HPC applications by presenting the Spark-MPIintegrated platform for bringing together Big Data analytics,HPC scientific algorithms and deep learning approaches fortackling new frontiers of data-driven discovery applications.The approach was validated with three MPI projects (MPICH,MVAPICH and Open MPI) and established a basis for advanc-ing the Spark-MPI interface towards the Exascale platformusing the PMI-Exascale (PMIx) framework. Furthermore, this

direction aligns with a paradigm shift from data-intensiveprocessing pipelines towards the fifth paradigm of knowledge-centric cognitive applications. Within the context of newapplications, Spark-MPI aims to provide a generic front-endfor distributed deep reinforcement learning platforms on HPCclusters. As a result, the Spark-MPI platform represents a triplepoint solution located at the intersection of three paradigms.

REFERENCES

[1] T. Hey, S. Tansley, & K. Tolle (eds.), The Fourth Paradigm: Data-Intensive Scientific Discovery, Microsoft Research, 2009

[2] V. Sarkar et al., “Synergistic challenges in data-intensive science andexascale computing, ” DOE ASCAC Data Subcommittee Report, 2013

[3] K. Kleese van Dam, T. Proffen and C. Tull, “Computing and datachallenges @ BES facilities,” BES Exascale Requirements Review, 2015

[4] Y. LeCun, Y. Bengio and G. Hinton, “Deep learning,” Nature, Vol. 521,2015

[5] V. Mnih et al, “Human-level control through deep reinforcement learn-ing,” Nature, Vol. 518, 2015

[6] D. Hassabis, D. Kumaran, C. Summerfield, and M. Botvinick,“Neoroscience-inspired artificial intelligence,” Neuron, Vol. 95, 2017

[7] R. Ackoff, “From data to wisdom,” Journal of Applied Systems Analysis,Vol. 16, 1989

[8] Y. Low et al., “GraphLab: a new framework for parallel machinelearning,” UAI, 2010

[9] A. Agarwal et al., “An introduction to computational networks andComputational Network Toolkit,” MSR-TR-2014-112, 2014

[10] M. Abadi et al., “TensorFlow: large-scale machine learning on hetero-geneous distributed systems,” Preliminary White Paper, 2015

[11] A. Nair et al., “Massively parallel methods for deep reinforcementlearning,” ICML, 2015

[12] G.Fox et al., “Towards an understanding of facets and examplars of BigData applications,” In Proc. of 20 Years of Beowulf, 2014

[13] Intel, the HDF Group, Cray and EMC, “Fast Forward Storage and IO,”Final Report, 2014

[14] M. Zaharia et al., “Resilient distributed datasets: a fault-tolerant abstrac-tion for in-memory cluster computing,” NSDI, 2012.

[15] M. Zaharia et al., “Descretized streams: fault-tolerant streaming compu-tation at scale,” SOSP, 2013

[16] “MPI: A Message-Passing Interface Standard,” Version 3.1, MessagePassing Interface Forum, 2015

[17] W. Gropp and E. Lusk, “An abstract device definition to support the im-plementation of a high-level point-to-point message-passing interface,”Preprint MCS-P343-1193, Argonne National Laboratory, 1994

[18] S. Ptluri et al., “Efficient inter-node MPI communication using GPUDi-rect RDMA for InfiniBand clusters with NVIDIA GPUs,” ICPP, 2013

[19] Jeffrey M. Squyres and Andrew Lumsdaine, “The Component Archi-tecture of Open MPI: Enabling Third-Party Collective Algorithms“,Component Models and Systems for Grid Applications, pp. 167-185,DOI 10.1007/0-387-23352-0 11

[20] Edgar Gabriel, Graham E. Fagg, George Bosilca, Thara Angskun, JackJ. Dongarra, Jeffrey M. Squyres, Vishal Sahay, Prabhanjan Kambadur,Brian Barrett, Andrew Lumsdaine, Ralph H. Castain, David J. Daniel,Richard L. Graham, and Timothy S. Woodall, “Open MPI: Goals,Concept, and Design of a Next Generation MPI Implementation,” Pro-ceedings, 11th European PVM/MPI Users’ Group Meeting, Budapest,Hungary, pp. 97-104 September 2004

[21] Pavan Balaji et al., “PMI: a scalable parallel process-managementinterface for extreme-scale systems,” In: Keller R., Gabriel E., Resch M.,Dongarra J. (eds) Recent Advances in the Message Passing Interface.EuroMPI 2010. Lecture Notes in Computer Science, vol 6305. Springer,Berlin, Heidelberg

[22] R. Hegerl and W. Hoppe, “Dynamische Theorie der Kristallstrukturanal-yse durch Electronenbeugung im inhomogenen Primarstrahlwellenfeld,”Phys. Chem. 7, 1970

[23] J. M. Rodenburg et al.,“Hard-X-Ray Lensless Imaging of ExtendedObjects,” Physical Review Letters 98, 2007

[24] P. Thibault et al.,“High-Resolution Scanning X-ray Diffraction Mi-croscopy,” Science 321, 2008

[25] P. Thibault et al.,“Probe retrieval in ptychographic coherent diffractiveimaging,” Ultramicroscopy 109, 2009

[26] K. Giewekemeyer, “A study on new approaches in coherent x-raymicroscopy of biological speciments,” PhD thesis, Gottingen series inx-ray physics, Volume 5, 2011

[27] V. Elser, “Phase retrieval by iterated projections,” J. Opt. Soc. Am. A,2003

[28] D. R. Luke,“Relaxed average alternating reflections for difraction imag-ing,” Inverse Problems 21, 2005

[29] S. Marchesini et al., “SHARP: A distributed, GPU-based ptychographicsolver,” LBNL-1003977, 2016.

[30] J. Kreps, N. Narkhede, and J. Rao, “Kafka: a Distributed MessagingSystem for Log Processing,” NetDB, 2011

[31] A. Vishnu, J. Manzano, C. Siegel, and J. Daily, “User-trasparent dis-tributed TensorFlow,” arXiv: 1704.04560, 2017

[32] F. N. Iandola et al., “FireCaffe: near-linear acceleration of deep neuralnetwork training on computer clusters,” arXiv: 1511.00175, 2015

[33] A. A. Awan, K. Hamidouche, J. Maqbool Hashmi, and D. Panda, “S-Caffe: co-designing MPI runtime and Caffe for scalable deep learningon modern GPU clusters,” PPoPP, 2017

[34] A. Sergeev and M. Del Balso,“Horovod: fast and easy distributed deeplearning in TensFlow,” arXiv: 1802.05799v3, 2018

[35] A. Gibiansky and J. Hestness, https://github.com//baidu-research/tensorflow-allreduce, 2017

[36] P. Patarasuk and X. Yuan,“Bandwidth optimal all-reduce algorithms forclusters of workstations,” J. Parallel Distrib. Comput.,69:117-124, 2009

[37] G. Litjens et al.,“A survey on deep learning in medical images analysis,”arXiv: 1702.05747, 2017

[38] Y. Li, “Deep reinforcement learning: an overview,” arXiv: 1701.07274,2017

[39] X. Lu et al.,“Accelerating Spark with RDMA for Big Data processing:early experiences,” IEEE 22nd Symposium on High-Performance Inter-connets, 2014

[40] A. Gittens et al.,“A multi-platform evaluation of the randomized CXlow-rank matrix factorization in Spark,” IPDPS, 2016

[41] M. Anderson et al.,“Bridging the gap between HPC and Big Dataframeworks,” VLDB, 2017

[42] N. Malitsky,“Bringing the reconstruction algorithms to Big Data play-forms,” NYSDS, 2016

[43] P. Moritz et al., “SparkNet: training deep networks in Spark,” ICLR,2016

[44] C. Noel, J. Shi, and A. Feng,“Large scale distributed deep learning onhadoop clusters,” 2015 (unpublished)

[45] L. Yang, J. Shi, B. Chern, and A. Feng, “Open Sourcing Tensor-FlowOnSpark: Distributed Deep Learning on Big-Data Clusters,” 2017(unpunlished)

[46] Jason Dai and Radhika Rangarajan, “BigDL: Bringing Ease of Use ofDeep Learning for Apache Spark,” Spark Summit, 2017

[47] MPI Sessions Working Group, https://github.com/mpiwg-sessions/sessions-issues/wiki

[48] Ralph H Castain, David G. Solt, Joshua Hursey, and Aurelien Bouteiller,“PMIx: Process Management for Exascale Environments,” Proceedingsof the 24th European MPI Users Group Meeting (EuroMPI/USA 2017),Chicago IL, USA, Sept 24-28, 2017, 14:114:10

[49] Ralph H Castain, David G. Solt, Joshua Hursey, and Aurelien Bouteiller,“PMIx: Process Management for Exascale Environments,” extend ver-sion to be published in Journal of Parallel Computing

[50] Artem Polyakov, Joshua S. Ladd, Boris I. Karasev, “Towards Exascale:Leveraging InfiniBand accelerate the performance and scalability ofSlurm jobstart,” presented in Slurm Booth at Supercomputing 2017

[51] M. Jette and M. Grondona, “SLURM: a simple Linux utility for resourcemanagement,” UCRL-MA-147996 Rev 3, 2003

[52] K. V. Vavilapalli et al.,“Apache Hadoop YARN: yet another resourcenegotiator,” SoCC, 2013

[53] B. Hindman et al.,“Mesos: a platform for fine-grained resource sharingin data senter,” NSDI, 2011

[54] K. Hightower, B. Burns, and J. Beda, Kubernetes: Up and Running Diveinto the Future of Infrastructure, O’Reilly Media, 2017

[55] V. Mnih et al.,“Asynchronous Methods for Deep Reinforcement Learn-ing,” ICML, 2016