2004 lEEE/PES Transmission 8 Distribution Conference 4% Exposition: Latin America 1

Parallel Processing in a Cluster of Microcomputers with Application in

Contingency Analysis Balduino, L., and Alves, A.C.B.

Abstract - This work is about the use of a microcomputer network as a parallel processing environment. In order to allow the communication, the synehronization and the distribution of the tasks among processes, the PVM (Parallel Virtual Machine) and MPI (Message Passing Interface) systems, were used. These computational systems allowed for parallel processing in the area of electric power systems, namely the contingency analysis. This application, usually found in Energy Management Systems, requires the use of a large quantity of resources in a short time. The aim i s to analyze thousands of disturbance scenarios, in a preventive mode in real Lime. Two parallel programming paradigms classified as MastedSlave, in the synchronous and asynchronous modes, were implemented. Case studies with sequential and parallel implementations were carried out to validate the algorithms used to analyze elaborated contingencies, including the use of data from the Interconnected Brazilian Electric Power System, allowing the analysis of single and multiple disturbances.

Index Terms - Contingency Analysis, Energy Management Systems, Static Security Analysis, Parallel Programming.

I. Introduction ince the appearing of the first generation of computers there is a constant aspiration in increasing their process capacity, There is also a vast need to perfom the greatest

possible number of tasks in the smallest period of time and cost. Several findings contributed meaningfully for that, such as, faster processors, memory hierarchy, bigger and more efficient disks, parallel computers, etc.

The development of the computer networks also contributed a lot to attend the necessity of faster processing of information, for with them tools even more sophisticated that allow not only peripheral sharing, but the communication among applications as well, appeared. It is possible, this way, to transform a computer network into a multiprocessor. The basic idea of the parallel programming is to divide an

S

This work was supported in part by the CAPES. The authors thank by the financial support received from the Program of Development and Research of the Brazilian Electric Sector, Federal Law 9.991, in particular to the local electricity company, CELG.

Liliane Balduino obtained her M.Sc. Degree in Electrical and Computer Engineering in the Postgraduate Program of the School of Electrical and Computer Engineering of the Federal University o f Goils, Brazil (e-mail: lilibalduino@bol.com.br).

A.C. Baleeiro Alves is with Schoot of Electrical and Computer Engineering of the Federal University of Goiis since 1985, Brazil (e-mail: baleeiro@eee.ufg.br). He is a150 with Catholic University of G o i k

application into small parts slightly independent with work in different machines of the network, which they eventually communicate with each other in order to exchange information. Two systems of parallel processing in computer network, highly spread nowadays, are the PVM and the MPI.

Among the applications of the systems of parallel processing, those which need to be executed in real time or that show restriction in execution time are detached. The Energy Management Systems (EMS) have functions of intensive computing which require the execution of thousands of disturbance scenarios, from the analysis of the electric network in the intact condition (base-case load flow), in the preventive mode. The analysis of contingencies of electric networks is one of those hnctions that demand the solution of thousands of nonlinear algebraic equations in an interval of time in the order of milliseconds. The ideal is that those preliminary studies provide the operator, and the own computing system, operational alternatives for the elimination of violations of equipment limits, or the search for a secure point for the operation. For the Energy Management Systems, gaining time in the execution of analysis of contingencies with the use of parallel processing, can, for example, facilitate the introduction of improvements in the mathematical model of the power system, or even improve the process of choice of operative decisions.

This article is organized the following way. Firstly, the programming systems are presented and parallel processing used. After that, a description of the structure of the modern EMS is done and also a brief analysis of its tendencies is done. The mathematical model of the analysis of contingencies is presented and discussed. A description of the parallel implementations is made, followed by the presentation of tests and results. Conclusions are extracted from database of power systems of the real world.

11. Parallel Processing in Microcomputer Network In the present time, with LAN (Local Area Network) and

the systems of parallel programming, the development of application with relative “parallelism grade” became natural. Due to the dedicated parallel architecture, with the nCube, the parallel processing in a cluster of microcomputers (i.e., group of interconnected computers able to communicate during the execution of tasks) presents the limitation of the simultaneous communication. For applications in which the quotient runningicommunication is high (i.e., “coarse grain”), excellent gains in speedups can be [2].

0-7803-8775-9104/$20.00 02004 IEEE 285

2

Depending on the intrinsic characteristics of the desired parallelism of the algorithm and the chosen programming paradigm, the parallel processing in networks will be efficient r2931.

c5 -

Fig. 1: Basic configuration of a cluster.

The clusters are interesting solution for the research centers and organizations to obtain a high performance processing in a low cost comparing to the cost of a supercomputer. In a software viewpoint, each host must have a network operational system, for example, UNIX FREEBSD, which manages the work of I/O and the basic communication through the network. In order to allow the communication among the hosts, as well as the distribution and coordination of tasks in the cluster, each host should also have a platform that provides such services in a level of suitable abstraction. Representative example of this kind of platform PVM [5] and the MPI 161, are pointed out, which are implemented through libraries of accessible functions to the parallel programs [2]. These systems have routines in high-level language that are easy to be used by the programmers.

111. Parallel Programming Tools With the purpose of giving an environment of parallel

programming in distributed memory systems, some routines were developed [l l] , where many of them are extension of existing procedure languages, FORTRAN and C being the most used. Such routines with the addition of PVM and MPI libraries enable an environment of message exchanges.

A . parallel Virtual Machine - PVM

The PVM is a software system that enables computing environments to work as if they were paralIel machines. The environment of the PVM started in 1989, in the Oak Ridge National Laboratory, and it has been improved since then. The PVM allows the programming of applications in a network constituted of heterogeneous machines that brings to the programmer the impression of being a unique big parallel computer. Other principles taken into consideration were of portability, scalability and dynamic configuration [5].

The PVM system is compounded of two parts. The first part is a daemon called pvmd, which runs in all the machines

~

286

that compose the cluster. When a user wants to process an application, he executes the pvmd in one the computers of the cluster, and then, starts pvmd in the other machines that compose the virtual machine. Libraries of interface of the P V M compound the second part of the system. These libraries contain routines, which are called in the application of the user as simple programming instructions. These routines allow the parallel application programmer make message transfers, allocate process in hosts of the cluster, coordinate tasks, modify the virtual machine, prevent computer or process failures, etc. PVM system allows heterogeneous computers in parallel machine.

E. Message Passing Interface - MPI

The standard MPI was concluded in 1994, joining definition of a group of standard routines for message passing, with syntax and semantic, both well defined [6].

The MPI was projected to execute in different machines efficiently. In this aspect, only the logic work of operation is defined. So, many implementations of libraries of the MPI standard have been developed, some of them being owners and some of free code, in a way that the developers may optimize the code using the specific characteristics of each machine.

In the model of programming with MPI, a parallel application consists of one or more processes that communicate among themselves through the routines for sending and receiving messages [6] . In the majority of implementations using MPI, a fixed number of processors is created statically, and only one process is allocated for each processor in the system. This reflects in the initial objective of the MPI, which consists in providing support for the programming of totally parallel computers. In spite of not being a rule, each one of these processes can execute different programs, which allows them to classify systems MPI as MPMD (Multiple Program Multiple Data) [Z, 61. In the present work the public domain MPICH version was used.

C. PVM and MPr Comparisons

Historically, the targets of the project of the parallel virtual machine (PVM) and the message passing interface (MPI) were different.

The development of the PVM started in the academic center for a distributed computing in heterogeneous environment research, while the MPI was created to be the standard of message passing in total parallel processors. The MPI was developed by experts in high performance computing as much in industry as in scientific group. The table I resumes the main differences and similarities between the two systems in inherent aspects to the parallel programming [5,61.

From the programmer viewpoint there is few difference behveen the two systems. For instance, the PVM needs additional instructions to initiate and to pack a message before sending to, while the MPICH not needs to.

3

TABLE I COMPARATIVE! SQUARE BETWEEN PVM AND MPI

tn the following section it is presented a description of the environment of the modern Energy Management System.

IV. Energy Management System Energy Management Systems (EMS), or simply, Energy

Control Centers, are structures mainly composed by qualified staff, hardware and software, besides modern data acquisition and transmission systems. These centers are responsible for the electrical system management, monitor, and control of a concession area, which can cover fiom one specific region to the whole interconnected system. The National Operator System (IS0 agent in the Brazil), which holds a large electric network of regional control centers, is responsible for executing the electrical system transmission and generation operation control and coordination activities on the National lnterconnected System.

The EMS, besides the control and supervisory system called SCADA - Supervisory Control and Data Acquisition - also have computational applications capable of processing data received fi-om the electrical system and picture its operational conditions [7].

Among many of the programs executed in the EMS, deserves prominence the electric network analysis hc t ions , which objective is monitoring the electrical system current operation, providing the operator a reliable estimate of the system state. These functions list the occurrence of undesirable operative conditions and present control strategies that allow changing the operation point to a new operative condition, a secure condition [9].

Due to the technological enhancement in computers (such as, open systems, downsizing and distributed processing), and the demand of an electrical system dimension and complexity increase, there is a worldwide tendency of the electric network analysis function to be execute in real time in the EMS, and the control actions being made and automatically done by the hardwarehftware with minimum human interference.

Low cost machines that offer a reliable and fast execution option, executing applications in fieeware platforms (such as LrNUX), are attractive alternatives for the demanding software and hardware requirements other than the expensive workstations.

Static Disturbance Analysis in EMS

Disturbance analysis execution involves intensive calculus that can be a high time consuming process 13, 81.

In a power system, a list of disturbances may be composed of individual lines outages (single disturbances) or multiple transmission lines outages (multiple disturbances).

Actions can be made in order to keep the system away from undesirable operation conditions due to any of the disturbances listed occurrence. These are known as preventive measures. In order to predict any harmful occurrence to the system operation, the operator, based on the software and hardware, must know in advance the seventy of each listed disturbance. Ln the literature this issue is known as contingency analysis.

If the electrical system disturbance analysis show through severity indexes that some listed disturbances compromise a secure operation of the transmission system, operative decisions can be made (involving either an operator or the regional control center), such as increase electrical system generation in a specific area or even transmission lines switching, in order to achieve a secure operation point. For example, nowadays, FACTS technology, Flexible AC Transmission System, allow controlling the electric network impedance with short time response.

The mathematical modeling of the disturbances analysis can be presented in three steps [4]:

The steady state load flow calculation of the intact electric network, known as base-case; h e analysis of the complete disturbance list in order to identify the most critical cases. Among these cases are the “islanding” (i.e., disconnection of the system) and also the cases where severe limits violations can lead to electric network operation alert and emergency state (the violations types that are normally calculated include bus voltage magnitudes and line power flows); Disturbances evahation, which is a most precise contingency analysis. Starting from a smaller list of the critical cases organized in decreasing order of severity indexes, the evaluation does the exact load flow calculation up to the converging point.

The calculus referred to in 1, operate with sparse matrices [ I 11, and iterative algorithms based on Newton method, sometimes of hard convergence depending on operative condition of the system. The screening referred to in 2, corresponds to the fast execution algorithms processing that provide approximate results. These methods are known as disturbance selection or screening. The results got from the selection allow making a small list of disturbances, which will be passed to step 3. Steps 1, 2 and 3 are mathematicalIy presented bellow.

Since xo is the vector of the electric network control variables and yo the vector of the dependent variable, both of them being related to the load flow, the calculation of the state and the line power flows of the electric system can be expressed according (l), designated by base-case [4].

g(x, y ) = 0 : algebraic equations of steady-state power flow model system;

287

A

h(x ,y ) 2 0 : operational limits of the system and of the equipments;

S : set of normal states of the base-case Ioad flow. The upper index '0' on the variables means intact

condition. If a list of m contingencies is given, in a way that the

changing of the electric network configuration is known for each contingency, the contingency analysis problem can be expressed according to (2).

0

g"(xV,y") = 0

h"(x",yV) 2 0

2, y" E sv

x", yv E s" (2)

S" : set of normal states of a contingency v . For each contingency, with the target of increasing the

efficiency of the calculations, the solution of the equation g(x, y ) = 0 uses the previous states and the sparse triangular factors of the matrixes obtained by solving (1).

The Fig.2 illustrates how the subproblems of the contingency analysis are related during the solution.

Fig. 2: Decomposition of the contingency analysis into subproblems.

h the analysis of the contingencies were implemented direct methods based in compensation and actualization of triangular factors for active {line power flows) and reactive (bus voltages) violations, considering single and multiple contingencies of branches (lines and transformers). In the analysis of the line power flows violations, for single contingencies it was used the method 1PB (post- compensation), for multiple contingencies till four simultaneous outages it was used the method 1PB (mid- compensation). For the simulation of more than four simultaneous contingencies it was used the actualization of triangular factors [3, 111, For the bus voltage magnitudes violations, with single contingency, the method 1P - lQ (post-compensation) was used, for multiple contingencies till four simultaneous exits it was used the method 1P - IQ (mid- compensation). For simulations of more than four simultaneous contingencies it was used the method of actualization of triangular factors. The calculation of the base- case load flow is made with the fast-decoupied method [4].

V. Contingency AnaIysis in Parallel Progamming Environments

The elaborated and treated programs in this article have specific routines, which allow several contingencies to be distributed to various processors and then be executed with significant saving of time.

In the elaboration of the programs it was used the model of master/slave programming. In this model, the master process reads the input data in the disks and transfer for the slave processes to execute (including the master process) the essential serial part of the program (calculus of the load flow) simultaneously. Once this stage is concluded, a new one starts, which is a typical parallel computing, where, each process receives the tasks for the execution of the intensive calculus (screening calculus). The results of this stage are reported to the process-0 (or master process), which makes the ordering of the contingency list, through the index of severity and sands new tasks for all the processes (slaves) execute. At this point it starts a new stage of parallel computing, in which each process executes intensive calculus of evaluation of the resumed list of contingencies. The result of each sIave process is managed in the process-0, where the output of the results is prepared.

A characteristic ofthis model is that the management is made by the process-0, and the intensive calculation in parallel (selection and evaluation) constitute in only one program that is executed in at1 the processor (including the process-0) for a portion of distinct data (data decomposition) equally. Only the master process accomplishes the operations of WO. Therefore, it is observed that the master doesn't become idle, for this process participates of the concurrent programming and, therefore, it contributes to increase the efficiency of the model (master/sIave).

The static contingency analysis presents independence among the tasks and it can be considered of high granulation. For that reason, it shows to be very suitable to the parallel execution with the PVM and the MPI. Each case of contingency to be processed is equivalent to the calculation of a load flow and needs the real state of the system (state vector) basically.

The Fig, 3 illustrates how the general execution of the calculation of analysis of contingencies in a network parallel processing environment is accomplished. In this figure, the symbols 's' and 'r' symbolize, data sending and receiving, respectively. In the blocks there are only two processes, the master (on the right) and a slave (on the left).

The implementation of the screening of contingencies and the evaluation of the resumed list, which are the effectively independent parts of the calculus, were decomposed in tasks of high granulation, where each host processes one contingency at a time, with the options referring to the communication of the synchronous and asynchronous mode. After that, the main aspects of these models of parallel programs and the way that each one was implemented were described.

288

5

Data lnpvt

Calculum of the calculum at tne

Parallel Processing screening 1

Results

Reordering the contingency list

4 1 Parallel Processing

Evaluation Of reduced list bS Results Resiilts rd Fig. 3: Blocks diagram of calculus of analysis of Contingencies in a parallel environment, representrng only two processes.

In the synchronous mode, the tasks are distributed among the hosts equally. In this mode it is presumed that the times of execution of the contingencies are similar. Actually, it doesn’t always happen. However, if the difference of the time were small, it can be worthwhile to use the synchronous mode.

In the contingency analysis through the synchronous processing, the hosts have a complete list of contingencies. For each processed case, the host stores the obtained results (for example, severity indices) locally in vectors. In the end of the analysis, the hosts send the results of the process-0.

In the asynchronous mode, each processor withdraws cases of the pool of contingencies managed by the process-0. Tne process-0 is responsible for coordinating the execution, distributing tasks, receiving flags from the hosts that concluded their cases, aflowing the finalization and, yet, processing cases. Initially, all the hosts are ready to analyze. As soon as a host finishes its task, it requires a new case for analysis. At this moment, the process-0 can be calculating contingency, verifying the buffer or attending a solicitation of another host. For an effect of comDarison of the modes of communication and distribution of tasks, the Fig. 4 illustrates two diagrams of execution of the calculus of the load flow, screening, ordering of the list of contingencies and evaluation of the list of the more severe contingencies, for three processes.

~

289

. asyrdmr”rmde Ordemg A ” r u p &

Fig. 4: Comparison of the profiles of execution of the synchronous and asynchronous mode for concurrent applications of contingency analysis.

The Fig. 4 shows that the screening and evaluation of contingencies are processed in parallel, while the base-case load flow is processed in a redundant way and the ordering of the list of contingencies is affected in a serial way. In the synchronous case, it is observed the overhead introduced by the waiting so that the processes that work mark may conclude their analysis. In the asynchronous mode, in the contrary of the synchronous, it is observed a higher intensity of communication among the processes.

VI. Tests and Results The test systems are two real world electric databases that

have the following characteristics: (a) 810 buses, 1340 branches, with the analysis of a list of 1300 contingencies containing single and multiple contingencies, namely BR-810; (b) 486 buses, 533 branches, with 533 single and multiple contingencies, namely BR-486.

For the tested systems contingency evaluations of 50 cases were executed, classified as critical. In all the tests shown here, the speedup is the ratio between elapsed time for a host and the elapsed time for the parallel running.

The machines that compound the cluster have the following configuration: Pentium IV, 2.3 GHz, and 532 MB of RAM memory. The cluster is constituted of a switch and the transmission media is the Ethernet. The implementations were written using a public domain C compiler.

The presented Fig. 5 and 6 show the results of the system BR-810 for the PVM and MPI, comparing the synchronous and asynchronous modes of communication.

malo PVM

Nmber of hods

Fig. 5: Speedups for ER-810 database, synchronous and asynchronous modes using PYM.

6

System BR486 BR810

point out to the grid computation using CORBA, Java and Internet resources, in power systems applications. BR310

MPI 5,o 4cJ VIII. Acknowledgment

0 3P The authors gratefully acknowledge the contributions of e 8 ZP Synchroncrua Fabio Moreira Costa, JoQ de Oliveira Junior, Mariane

IP @P

L I RAoynchronow Sardenberg Gomes de Caiado Castro and Ana Claudia Marques do Valle. i 2 3 4 5

U 0 hadf low Screening Evaluation TotaI% 6.1 2.8 1 I .7 79.4 IO0 3.3 1.5 21.2 74 100

Nvnber of hOSts

Fig. 6: Speedups for BR-810 database, synchronous and asynchronous modes using MPI*

It is observed, from Figs. 5 and 6 that the synchronous mode with the PVM presented better paformance, comparing with MPI, in spite of a small difference. It is verified a tendency of growing of the speedup when the number of machine is increased. However, it is expected that the stabilization of the earning (speedup) happen for a bigger quantity of processors.

The table I1 presents the percentual time to execute the contingency analysis using only one machine. This table is presented to facilitate the analysis of the results.

TABLE I1 PERCENTUAL TIME OF THE CALCULUM PHASES

In this table it is observed that the calculus of the screening and the evaluation consumed longer periods of time, for they are the parts of evaluation calculus in parallel. It is verified that the calculus of evaluation is what consumes longer period of time, since the heavier calculus of the processing of the analysis of contingencies are concentrated in this stage. It is obvious that the performance of any parallel system computing is affected by the size of the problem among other factors.

In absolute values, for the electric network of 486 buses, the contingency analysis including all phases was executed in 159 milliseconds in one machine, while, with five machines, the same system was executed in only 55 milliseconds.

VII. Conclusions It is important to show after the obtained results, that the

use of the PVM with the synchronous mode reached the best results in most of the made tests for this kind of application.

This work confirms that it is possible and reliable to use parallel processing environment in computer network in situations that require a high performance processing, as it is the case of static contingency analysis of bulk power systems in real time applications.

The computational systems PVM and MPI showed robustness and liability. Both of them are relatively easy to program and also present similarities with the use of the libraries of programming. A dificulty that has to be projected refers to the grade of difficulty of depuration of the implementations during the programming. Future researches

IX. References [ I ] V. Kumar, A. Grama, A. Gupta and G. Karypis. "Introduction to parallel

computing - design and analysis of algorithms", New York The Benjamin Cummings Publishing Company, Inc, 1994, p. 597.

[2] R. Buyya. "High performance cluster computing", New Jersey: F'rentice Hall, 1999, p. 849.

[3] A. C. Baleeiro Alves. "Processamento distribuido aplicado i anllise de seguraya esatica de sistemas de energia elktrica", Campinas: FEE, UNICAMP, 1997. Tese (Doutorado) - Faculdade de Engenharia EICtrica, Universidade Estadual de Campinas, 1997. p. 208.

[4] A. J. Monticelli. "Fluxo de carga em redes eldtricas", Slo Paulo: Edgard Blicher, 1983, p.164.

[SI A. Geist., J. Dongarra., W. hang and V. Sunderam. "PVM: Parallel Virtual Machine, an user's guide tutorial for networked parallel computing". London: Library of congress cataloging-in-publication data. 1989.

[ 6 ] S. Snir, S. Otto, S. Huss-ledetman, D. Walker and J. Dongarra. "MPI: the complete reference", London: Library of Congress catalogin-in- publication Data, May 1996.

[7] R.B. Sollero et al. "Os centros de operaqlo M nova estrutura do setor eletrico". In: IV SIMPASE ~ Simpdsia de AutomaGiio de Sistemas EIptricos, 2000, IV SIMPASE Anais ... Informe ticnico BSB/02/07- GRUPO IV. Brasil.

[8] D. P. Pinto and J. L, R. Pereira. "Um m&do integrado para anilise de contingincias utilizando processamento paralelo". X Ci3.4. Congress0 Brmileiro de Automtiticu. 1994 -Rio de Janeiro- Brasil, p.179-184.

[9] G. P. de Azevedo, M. Costa e B. Feij6. "A estrutura da proxima geraqb de centros de controle de energia", In: X V S N P E E - Seminario Nacional de Producrio e Transmissdo de Energin Elktricu, 1999, XV SNPTEE. Anais ... Fozdo Iguacu - Brasil, p. 7.

[IO] J.C. Mendes, 0. R. Saavedra and S . Feitosa. "A paralIel complete method for real-time security analysis in power systems", Electric Power Systems Reseurch, 56,2000, pp. 27-34.

[I I] N. 0. Alvares. "Solu@o eficiente de sistemas lineares esparsos", Goiinia: EEEC, UFG, 2004. DissertaFEo (Mestrado) - Escola de Engenharia Elttrica e de ComputaqBo, Universidade Federal de Go&, 2004. p. 143.

X. Biographies Liliane Balduino was bom in Goilnia-GO, Brazll She obtained a Master's degree from the Federal University of G o i b (WG) in Electrical and Computer Engineering and a Bachelor's from the Salgado de Oliveira University (UNIVERSO) in, respectively, 2004 and 2000 She is currently an associate professor at the State University of Go& (UEG) Her research interests include parallel processing and numencal

analysis Antinio Char Baleeiro Alves was bom in Tebfilo Otoni-MG, B m i l He received a PhD degree from the State University of Campinas (UNICAMP) (1997), a Master's in Electrical Engineenng, from the Federal Univensty of

Uberlhdia (UFU) (1991) and pduated from UFG in 1983 He IS an Adjmct Professor in the School of Electrical and Computer Engineering at UFG. His special fields of interest included power systems and computatlonal mathematics

290