A service-oriented business performance evaluation model and the performance-aware service selection method

CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCEConcurrency Computat.: Pract. Exper. 2008; 20:1821–1836Published online 6 June 2008 inWiley InterScience (www.interscience.wiley.com). DOI: 10.1002/cpe.1320

A service-oriented businessperformance evaluation modeland the performance-awareservice selection method

Bo Liu∗,†, Yushun Fan and Shuangxi Huang

Department of Automation, Tsinghua University, Beijing 100084, People’sRepublic of China

SUMMARY

In service-oriented architecture, services and business processes are closely related and therefore theresearch on service-oriented business process (SOBP) attracts the attention of academia as well as in-dustry. Because of the loosely coupled, autonomic and dynamic nature of services, the operation andthe performance evaluation of business process meet some challenges, such as the definition of the keyperformance indicators (KPIs) and the alignment of business performance and IT performance. In thispaper, we address these challenges. First, the definition and the characteristics of an SOBP are presentedwith a motivating scenario. Then a service-oriented business performance evaluation model is described,which integrates the performance of strategy layer, business process layer, business activity layer, servicecomposition layer and IT infrastructure layer. The KPIs corresponding to each layer are also definedwith their formal representations. The qualitative and quantitative performance analysis methods, basedon the proposed model, are presented, respectively. Finally, the improved analytic hierarchy process isexplicated to calculate the correlation between different KPIs and select the most suitable service. Anonline-shopping example is taken to prove the soundness and the feasibility of the method. Copyright ©2008 John Wiley & Sons, Ltd.

Received 25 January 2008; Accepted 6 February 2008

KEY WORDS: business process; service-oriented architecture; performance evaluation; key performance indicator

∗Correspondence to: Bo Liu, Department of Automation, Tsinghua University, Beijing 100084, People’s Republic of China.†E-mail: [email protected], [email protected]

Contract/grant sponsor: National Natural Science Foundation projects of China; contract/grant numbers: 60674080, 60504030Contract/grant sponsor: National High-Tech R&D (863) Plan of China; contract/grant number: 2006AA04Z166Contract/grant sponsor: National Basic Research Program of China; contract/grant number: 2006CB705407Contract/grant sponsor: European Commission FP6 project; contract/grant number: FP6-033610

Copyright q 2008 John Wiley & Sons, Ltd.

1822 B. LIU, Y. FAN AND S. HUANG

1. INTRODUCTION

With the advent of new computing paradigms such as Grid [1], service-oriented architecture [2]and P2P computing [3], Web services are gaining momentum as key elements in enterprise in-formation systems. On building a loosely coupled services ecosystem, more and more researchesare focused on service composition, in which workflow and business process management haveshown great potential. There is an emerging tendency that combines Web service and businessprocess together [4–6]. Because of the loosely coupled, autonomic and dynamic nature of services,service-oriented business process (SOBP) represents some new characteristics and challenges withregard to operation mechanism and performance evaluation.Many studies have been dedicated to the performance evaluation of business system and IT

system. Liu et al. [7] proposed an evaluation system reflecting the synthetical performance of busi-ness processes. Zhu et al. [8] presented a performance evaluation system based on manufacturingprocesses, but lacked particular description on the criteria in each layer. Chen [9] analyzed work-flow performance through generalized stochastic Petri nets, yet needed further research on resourcedistribution and data share. Spooner et al. [10] described a performance-aware grid managementsystem that supported workflow scheduling using a multi-domain performance management infras-tructure. Menasce [11] discussed the quality of service (QoS) issues in Web services. Song andLee [12] introduced a performance analysis tool sPAC especially toward Web services, but onlytime properties were illustrated. Chen and Yang [13] depicted the verification and validation of gridworkflow.With its unique importance for business control and improvement, business performance eval-

uation has raised enormous interest in academia. However, most existing performance evaluationsystems focus on customers and markets in business domains without consideration of the rela-tionship with IT domains, thus resulting in the disjunction of business and IT systems. In addition,existing evaluation systems ignore the QoS and the analysis of relationship between different keyperformance indicators (KPIs), and the definitions of KPIs are normally limited to special domainswith low flexibility. In order to address these issues, a service-oriented performance evaluationmodel is proposed in this paper. It integrated business view and IT view, brought into service com-position layer (SCL) with the estimation of service performance, thus fitted to the real operatingsituation of enterprises.Based on the proposed evaluation model, the qualitative performance analysis method is de-

picted with an example derived from a practical enterprise information system. And the quanti-tative analysis method is represented through calculating the correlation between different KPIs.This paper utilizes an analytic hierarchy process (AHP) for quantitative analysis. The AHP is apowerful and flexible multi-criteria decision-making method for complex problems and has beenused in many governmental and industrial applications. However, its applications in business pro-cess domain are seldom [14,15]. In this paper, an improved AHP method is proposed in order toanalyze the correlation and select services according to their synthesized performance. Further-more, an online-shopping example is taken to illustrate the correctness and the feasibility of thismethod.The remainder of the paper is organized as follows. The definition and the characteristics of

an SOBP are proposed with an example in Section 2. Then we expatiate on the performanceevaluation model in Section 3, including the definition of KPIs in each layer and their formal

Copyright q 2008 John Wiley & Sons, Ltd. Concurrency Computat.: Pract. Exper. 2008; 20:1821–1836DOI: 10.1002/cpe

A SOBP EVALUATION MODEL 1823

representations. Based on the performance evaluation model, Section 4 gives qualitative anal-ysis with an example extracted from an enterprise information system. Section 5 gives quan-titative analysis using improved AHP method to calculate the correlation of KPIs in differentlayers and select services. Finally, conclusions and directions for future work are presented inSection 6.

2. DEFINITION AND CHARACTERISTICS OF AN SOBP

Service science, as a new field, has become the focus in recent years, but there still lacksa uniform definition of the service. In this paper, a service is defined as an IT-enabled orIT-innovated functionality involving certain business process or activity, which is offered by aprovider.Because of the nature of services, such as loosely coupling, coarse granularity, access trans-

parency, platform independency and business orientation, business process in service-oriented en-vironments also presents some new characteristics as follows:

1. Services could compose a business process.2. Business process could be encapsulated into a service.3. Multiple processes interact with events/messages and share common resource or data.4. The processes change dynamically along with the change of services. It requires ensuring the

usability of services and selecting service components in real time, which also results in thedifficulty in evaluating the performance of business processes.

Based on the above characteristics, we proposed the definition of SOBP.SOBP is the business process partly or totally executed by the computers automatically in service-

oriented environments, partial or entire activities in the business process are completed by servicesin Network. In other words, SOBP is a composition of Web services for the purpose of specialtasks.Taking an online-shopping process as an example, Figure 1 illustrates the scenario of a business

process in service-oriented environments. When a customer performs an online shopping, the orderwith customer information is submitted. The system checks whether the customer is registered ornot, if yes, continues to check the inventory, if not, gives an error report to the customer. Whenthe inventory is lacking, the customer is notified and is recommended to purchase other relatedgoods. While the inventory is enough, the system informs the customer to select the pay modeaccording to his/her preference, such as bank transfer, remitting, paying on delivery or e-Pay. Afterthe payment is confirmed, the goods will be delivered and the customer can choose either the acceptor reject mode.In this process, some activities are required to communicate with the customers, and some

activities work as services, requiring agents in Enterprise Service Bus to perform the correspondingfunctions. For the activity ‘check customer’, the agent queries the customer relationshipmanagement(CRM) system. Similarly, for the activity ‘check inventory’, the agent queries the enterprise resourceplanning (ERP) system. The functions of CRM and ERP systems are encapsulated into inter-enterprise services, which could be invoked by other systems.



Figure 1. Business process in service-oriented environments.

For payment service, suppose that the customer selects e-Pay to pay online, then the delivery modeis decided including home-delivery, mail, express and express mail service (EMS). The paymentservice and delivery service are all inter-enterprise services open to users out of the domain of theirproviders. Each service may have hundreds of providers; therefore, the agents need to choose themost suitable service with best performance.In traditional situations, ERP and CRM systems may be invoked from different portals and can-

not be integrated seamlessly. The payment and delivery activities also need to be completed bysearching service providers artificially. However, through encapsulating inter-enterprise servicesincluding legacy systems such as CRM, ERP, supply chain management (SCM) and product datamanagement systems, and through developing inter-enterprise services covering physical resources,information resources and service resources, the issues of information integration, process integra-tion and application integration are resolved.It can be seen that the operation mechanism of SOBP is quite different from that in traditional

situations. The execution of business processes needs to dynamically select and invoke services;hence, the performance of service must be considered when evaluating business performance, andthe relationship between service performance and business performance should be analyzed. Thefollowing sections address these issues.



3. PERFORMANCE EVALUATION MODEL FOR SOBP

The performance of business process includes several aspects: such as time, cost, quality, reli-ability, etc. The evaluation of single performance indicator is obviously unilateral. Performanceevaluation should be toward multi-indicators synthetically [16]. Hence, the key problem is to de-sign a performance evaluation model by which multi-indicators and their relationships could bereflected.

3.1. Performance evaluation model

Figure 2 shows a service-oriented performance evaluation model. The business system and the ITsystem are divided into five layers. Every layer has its corresponding KPIs.

1. The bottom layer is IT infrastructure layer (ITL), and the corresponding KPIs are through-put, delay, bandwidth, etc. that reflect the performance of the network, operating system,instruments, and so on.

2. SCL is used to composite required services from thousands of services according to theuser’s requirements. The requirements include functional and non-functional aspects; hence,this layer’s KPIs contain functional indicators (used for estimating whether the function ofthe service matches user’s demand) and non-functional indicators (used for estimating theQoS).

3. Business process layer (BPL) and business activity layer (BAL) are the core layers. Theircorresponding KPIs, process-related indicators and activity-related indicators are discussed inSection 3.2. For normal tasks, we do not need to select or invoke services, so the indicators inBAL map to ITL directly, whereas for service node, the indicators in BAL map to SCL andthen to ITL.

4. The top layer business strategy and design layer (SPL) faces end users and managers. Strategicgoals vary from user to user, including lower business costs, increased adaptability, flexibilityand efficiency, lower systems implementation risks, better governance and compliance, bettercustomer satisfaction, etc.

The two bottom layers are from the IT view, whereas the others are from the business view. Eachlayer maps to neighbor layer according to KPIs’ mapping.Most existing performance evaluationmodels merely consider the mapping from business strategy

layer to BPL, or merely consider the performance of Network in IT layer, thus resulting in thedisjoint of business and IT systems. The evaluation model in Figure 2 takes into account bothsystems synthetically and introduces SCL to present the particularity of services sufficiently. Thequalitative analysis of the relationship between business and IT systems is discussed in Section 4,and the quantitative analysis of their relationship is given in Section 5.

3.2. Performance evaluation indicator

Because of the variety of business processes, Figure 2 does not list all the indicators in everylayer, but depicts the general evaluation model based on which the users could customize specificevaluation model suitable to their own business. As far as ITL and SPL, related research is mature



Figure 2. Service-oriented performance evaluation model.

to some extent. The functional indicators of services are determined by a specific service. The QoShas become a hotspot at present [11,17]. Owing to the page limitation of this article, we mainlydiscuss the indicators in BPL and BAL.Process-related indicators and activity-related indicators have close relation. Taking [18] for

reference, the concrete indicators and their calculating methods are listed in Table I.In order to represent the KPI more precisely, the formal representation of the KPI is proposed as

follows:AKPI can be represented as a variable K (t, �) ∈KPI, where t denotes time,�= {SP,BP,BA, SF,

QoS, IT} infers that K is strategic goals, process-related indicators, activity-related indicators, func-tional indicators of service, QoS and indicators in ITL, respectively, and KPI is the collection ofentire KPIs.The computing function of KPI K (t,�) is then defined as F(K1, K2, . . . , Kn), which is a function

of other KPIs in the same layer or lower layers with K ; n ∈ N .For instance, ‘the response time of the service= delay time + process time’ is presented as

K (t,QoS) = F(K1, K2) = K1(t,QoS) + K2(t,QoS)The relationship between different KPIs is defined as correlation w(Ki , K j ), where i, j ∈ N ,

w ∈ [0, 1]. The value of w means the degree to which Ki is affected by K j . In detail, w = 0 denotesthat Ki and K j are irrelevant, while w = 1 implies that Ki only has relation with K j .In addition, R(Ki ) represents the set of indicators whose correlation with Ki is bigger than zero

R(Ki ) = r(Ki , K j1) ⊕ r(Ki , K j2) ⊕ · · · r(Ki , K jm )



Table I. Process-related indicators and activity-related indicators.

Process-related indicators Activity-related indicators

Process cost equals the sum of activity cost for theexecuted activities. PC(t)=∑n

i=1ACi (t) where PCis the process cost, AC the activity cost and n thenumber of activities implemented in this process

Activity cost is the cost incurred when a task is exe-cuted. It equals the sum of enactment cost and realiza-tion cost. Enactment cost refers to the cost associatedwith the workflow system deployment, operation, main-tenance and monitoring. Realization cost corresponds tothe cost associated with the runtime execution of thetask. AC(t) =EC(t) + RC(t) where AC is the activitycost, EC the enactment cost and RC the realization cost

Process time equals the sum of activity time for ac-tivities in key route. PT(t) =∑m

i=1ATi (t) where PTis the process time, AT the activity time and m thenumber of activities in key route

Activity time equals the time an activity instance takesto be processed. It can be divided into two parts: delaytime and execution time. Delay time corresponds to theinstance queuing delay and the setup time of the instance,which are non-value-added time. Execution time is thetime an activity instance takes while being processed.AT(t) =DT(t) + ET(t) where AT is the activity time,DT the delay time and ET the execution time

Reliability of process. R(t) = 1− (SFR(t) +PFR(t))where R is the reliability of the process, SFR thesystem failure rate is the ratio of the number of timesa task did not perform for its users to the number oftimes the task was called for execution and PFR theprocess failure rate provides information concerningthe relationship between the number of times the statedone/committed is reached and the number of timesthe failed/aborted state is reached after the executionof a process

Reliability of activity. R(t)= 1 − (SFR(t) + AFR(t))where R is the reliability of activity, SFR the systemfailure rate is the ratio of the number of times a taskdid not perform for its users to the number of times thetask was called for execution and AFR the activity fail-ure rate provides information concerning the relationshipbetween the number of times the state done/committedis reached and the number of times the failed/abortedstate is reached after the execution of a task

Availability of the process is the probability that theprocess is available. AP(t) =UT(t)/TT(t) =UT(t)/UT(t) + DT(t) where AP is the availability of pro-cess, UT the UpTime is the total time the processhas been available during the measurement period,DT the DownTime is the total time the process hasnot been available during the measurement periodand TT the TotalTime is the total measurement time,which is the sum of UT and DT

Availability of activity is the probability that theactivity is available. AA(t) =UT(t)/TT(t)=UT(t)/UT(t) + DT(t) where AA is the availability of activ-ity, UT the UpTime is the total time the activity hasbeen available during the measurement period, DT theDownTime is the total time the activity has not beenavailable during the measurement period and TT the To-talTime is the total measurement time, which is the sumof UT and DT

Resource utilization. RU(t) =BT(t)/BT(t) + FT(t)where RU is the resource utilization, BT the busytime of the resource during the process executionperiod and FT the free time of the resource duringthe process execution period

Resource utilization. RU(t) =BT(t)/BT(t) + FT(t)where RU is the resource utilization, BT the busy timeof the resource during the activity execution period andFT the free time of the resource during the activityexecution period

Note: In this table, t means measuring time.



where r(Ki , K jk ) means that Ki is affected by K jk , and w(Ki , K jk )>0; m is the number of KPIsthat are related to Ki ; i, j,m, k ∈ N .It is clear that for each computing function FKi , f : FKi → R(Ki ), where f denotes the mapping

relationship between FKi and R(Ki ) and that R(Ki ) has transitivity. If r(Ki , K j ) and r(K j , Km) aregiven, ∃r(Ki , Km), where i, j,m ∈ N . Especially, if ∃Ki and R(Ki ) = �, then Ki is called atomicKPI, represented as Ki ∈KPI∗. For example, if the CPU queue length could not be influenced byother KPIs, then it is an atomic KPI.

4. QUALITATIVE ANALYSIS BASED ON THE PERFORMANCE EVALUATIONMODEL

In this section, we will give qualitative analysis of the relationship between business performanceand IT performance. Based on the above performance evaluation model, we take an enterpriseapplication suite (EAS) system in a certain company as an example to illustrate how to find thefactors that influence the business targets and what is the influence of IT equipment to businesstargets. This is a practical problem extracted from a project.The EAS system is an integrated platform that includes many modules such as ERP, CRM, SCM,

human resource management and Finance Management. After a period of operation, the users of theEAS system were not satisfied with this system because the response speed of the system decreasedsometimes. The company did not know why it happened and how to resolve it.In this situation, we utilized our performance model to analyze their problem (see Figure 3) .

Obviously, the main target in SPL is ‘customer satisfaction’, and the influence factor of it lies in‘process completion time’ in BPL. The processes of EAS system comprise many activities; hence,the ‘process completion time’ is divided into completion time of every activity in that process. Someactivities are implemented by calling inter-enterprise services from wide area network (WAN) andothers by calling inter-enterprise services provided by the function of inner information systems.The response time of these services influences the KPIs in BAL.ITL includes operating system layer, database layer, application server layer and network layer

from bottom to top. In this case, the ITL is expanded more concretely for the reason that we candefine the problem to a more elaborate extent.In the network layer, the ‘delay of WAN’ is the main reason that influences ‘response time of

inter-enterprise services’ and the ‘congestion of LAN (local area network)’ is the main reason thatinfluences ‘Response time of inner-enterprise services’. Through optimizing topology, expandingbandwidth and improving the performance of the router, the performance of network could beimproved.Moreover, the other influence factors contain load balancing and cache size in application server

layer, parameter configuration, structure design and SQL in database layer, and CPU frequency,memory capacity and hard disk space in operating system layer, etc.After the analysis of the log of the EAS system, it was found that the activities refer to search

and statistics always spend much time. That is to say, the ‘analysis time’ in BAL is the crucial KPI.According to the performance evaluation model, the service that influences ‘analysis time’ lies in‘analysis service’ in the ERP module; hence, ‘analysis service’ for searching and statistical analysisshould be analyzed further to shorten its response time. Then, from the KPIs related to ‘analysis



Figure 3. Performance analysis of the EAS system.

service’, Server 1 in ITL was found to be optimized. We modified the parameters of IBM DB2,which was a database installed on Server 1, optimized the SQL statements and increased the CPUfrequency of Server 1. Finally, the process completion time decreased by 21% in average, and theperformance of the whole EAS system was improved via the above measures.On the other hand, according to the performance evaluation model, the influence of IT equipment

to business targets is also defined clearly. For example, if Server 2 was damaged by virus, then whatwould be the result on business level? Through the relationship between KPIs, we found that the‘submit orders’ process and ‘stock in’ process were delayed because of Server 2, then the customerswho were submitting orders and the providers who were stocking goods into warehouses wereinfluenced.

5. QUANTITATIVE ANALYSIS BASED ON THE PERFORMANCE EVALUATIONMODEL

After qualitative analysis based on the proposed model, this section will focus on the quantitativeanalysis. The correlation of two indicators as mentioned above is an important measurement of therelationship of two indicators; hence, the process of calculating the correlation becomes a significantissue.



The AHP is one qualitative and quantitative method of the analysis on multi-objective decision. Itdecomposes a complicated problem into component elements, which are organized into a hierarchi-cal structure in terms of the relationships among component elements. The pair-wise comparison ofcomponent elements determines the relative decision element weights. Thus, an order on decisionweights of component elements can be worked out.Because the characteristics of AHP are suitable to the requirements of performance evaluation of

business processes, we employ the theory of AHP for the correlation analysis of KPIs in differentlayers, and furthermore for the selection of services.Traditional AHP method divides the decision-making system into three layers: the target layer,

the criteria layer and the plan layer.The improved AHP method is as follows. The target layer corresponds to the indicators (supposed

to be Ki ) measured by the managers. The criteria layer corresponds to a set of indicators influencingthe target indicator (i.e. R(Ki )), after the analysis we can achieve the weights of the indicators inthe criteria layer to the target indicators (i.e. correlation w(Ki , K j ), where K j ∈ R(Ki )), thusthe manager could obtain the importance degree and correlation of every KPI. The plan layercorresponds to concrete IT facility, optional services or various business activities. We do not usetraditional methods to calculate the weight of indicators in the plan layer to those in the targetlayer, but utilize concrete indicator values and differentiate quantitative indicators and qualitativeindicators and finally calculate the synthesized weights of each plan and support the decision ofmanagers.This section illustrates the mapping from the target layer to the criteria layer through a service

selection example. The improved AHP are as follows:Step 1: Identify the target indicator KA.Step 2: Define the related indicators in criteria layer according to R(KA) and build up the

hierarchical structure.Step 3: Build all the pair-wise comparison matrices in terms of the relative importance between

two indicators.Step 4: Estimate judgment matrix, compute the eigenvector and eigenvalue of the matrix, check

consistence of the matrix and calculate the synthesized weight of each indicator in the bottom layer.Step 5: Obtain the value of related indicators for the plan to be evaluated.Step 6: Normalize quantitative indicators and preprocess qualitative indicators.Step 7: Attain the synthesized weight of each plan through linear weighting method.Taking the online-shopping process in Figure 1 as an example, the calculation process of the

improved AHP is explained below.

5.1. Construct the AHP model

The main strategic target of this process is customer satisfaction. According to the evaluationmodel presented in Section 3.1, the calculation function and the relation function of ‘customersatisfaction’, we find the corresponding indicators that influence ‘customer satisfaction’ and buildup a hierarchical structure (see Figure 4) .The influence factors of ‘customer satisfaction’ contain many aspects; however, we just consider

three major factors in this paper, which are ‘completion time’ (B1), ‘total cost’ (B2) and ‘reliability’(B3). ‘Completion time’ is the process completion time in BPL. Compared with other activities,



Figure 4. The hierarchical structure for customer satisfaction.

the completion time of three activities (‘check inventory’, ‘delivery’ and ‘mail invoice’) playsan important role; hence, B1 is divided into C1–C3. ‘Total cost’ is the process cost in BPL. Inthis example, we normally consider product cost, delivery cost, inventory cost and mail invoicecost; hence, B2 is divided into C4–C7. ‘Reliability’ reflects the probability of error that occursduring the shopping process; hence, B3 mainly consists of ‘delivery reliability’ (C8) and ‘systemreliability’ (C9).Because the activities ‘check inventory’, ‘delivery’ and ‘mail invoice’ are all service nodes, the

indicators in BAL map to those in SCL directly. Product cost and inventory cost are external inputsfor this process, thus they are not expanded here. D1 maps to E1 and E2 in ITL further. ‘Systemreliability’ (C9) directly maps to ‘system failure rate’ in ITL.As shown in Figure 4, the hierarchical structure seems like a tree, in which the target layer A is

the top of the tree, B–E are the criteria layer and the atomic KPIs are the leaves of the tree.

5.2. Build judgment matrix and check consistency

The relative importance of factors adopt the 9-point scale, in which 1 denotes equal importance,3 denotes moderately more importance, 5 denotes strongly more importance, 7 denotes very strongly



more importance and 9 denotes extremely more importance. The values of relative importance aregiven by domain specialists.By using the function eig() of MATLAB6.5, the maximum eigenvalue �max is achieved. Then

the consistency index (CI) of the judgment matrix can be obtained as follows:

CI = �max − n

n − 1(1)

where n is the number of factors that had been compared.The consistency ratio (CR) of CI to the mean random consistency index RI is expressed as

follows:

CR= CI

RI(2)

where RI can be obtained via searching Table II provided by Saaty [19].It should be pointed out that if CI does not meet consistency’s requirement, judgment matrix

need to be constructed again.Taking aim at target A, judgment matrix A–B is constructed in Table III, where W means the

eigenvector. Calculate the maximum eigenvalue �max = 3.039, the consistency index CI = 0.019,CR= 0.033. Because CR<0.1, the judgment matrix A–B is consistent absolutely.

Table II. The value of RI.

n 1 2 3 4 5 6 7 8 9RI 0 0 0.58 0.90 1.12 1.24 1.32 1.41 1.45

Table III. Judgment matrix A–B.

A B1 B2 B3 W

B1 1 3 5 0.637

B2 13 1 3 0.258

B3 15

13 1 0.105

Table IV. Judgment matrix B1–C and B2–C .

B1 C1 C2 C3 W B2 C4 C5 C6 C7 W

C1 1 15

12 0.122 C4 1 2 3 5 0.496

C2 5 1 3 0.648 C5 12 1 2 2 0.245

C3 2 13 1 0.230 C6 1

312 1 1

2 0.115

�max = 3.004, CI = 0.002CR= 0.003<0.1 C7 1

513 2 1 0.144

�max = 4.132, CI = 0.433CR= 0.048<0.1



Table V. Judgment matrix B–C .

B B1 B2 B3C 0.637 0.258 0.105 W

C1 0.122 0 0 0.078C2 0.648 0 0 0.413C3 0.230 0 0 0.147C4 0 0.496 0 0.128C5 0 0.245 0 0.063C6 0 0.115 0 0.030C7 0 0.144 0 0.037C8 0 0 0.7 0.074C9 0 0 0.3 0.031

Taking aim at ‘completion time’ and ‘total time’, the judgment matrices of their sub-criteria arepresented in Table IV by using the same calculating method as matrix A–B.Similarly, we construct the judgment matrix B3–C . The synthetical judgment matrix of layers

C to B is depicted in Table V.The weight of indicators in layer D is equal to that in layer C

WD1 =WC1, WD2 =WC2, WD3 =WC3, WD4 =WC4, WD5 =WC6, WD6 =WC8

As far as the completion time of service ‘check inventory’, build judgment matrix similarly achieves

WE1 = 0.058, WE2 = 0.020

WE3 =WC9 = 0.031

Accordingly, the weight of each atomic indicator to target A is achieved

W = [WE1,WE2,WE3,WD2,WD3,WD4,WD5,WD6,WC5,WC7]T

= [0.058, 0.020, 0.031, 0.413, 0.147, 0.128, 0.030, 0.074, 0.063, 0.037]T

Through sequencing the weights, it is concluded that D2 has the most significant influence oncustomer satisfaction. Then the manager could utilize this result to achieve the target efficiently byimproving the indicator with the biggest weight. To sum up, the weight vectorW (i.e. the correlationof two indicators w(Ki , K j )) is calculated.

5.3. Select services

Having obtained the weight of the indicator, we can further utilize it for service selection.Firstly, the realistic values of the indicators in SCL could be acquired either by directly com-municating with service providers or by querying the corresponding service registration centersuch as UDDI. Then those original values should be preprocessed according to the followingmethod.



Quantitative indicator is normalized to a number between 0 and 1. It could be divided into twotypes:

1. Efficiency indicator (i.e. larger value is better) could be normalized by

ki =

⎧⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎩

1, Ki>Kmaxi

Ki − Kmini

Kmaxi − Kmin

i

, Kmaxi ≥Ki≥Kmin

i

0, Ki<Kmini

2. Cost indicator (i.e. smaller value is better) could be normalized by

ki =

⎧⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎩

0, Ki>Kmaxi

Kmaxi − Ki

Kmaxi − Kmin

i

, Kmaxi ≥Ki≥Kmin

i

1, Ki<Kmini

where Ki , Kmaxi and Kmin

i are the realistic value, maximum value and minimum value ofindicator i , respectively.

Qualitative indicator could be transformed to interval number by logic scoring of preferencemethod. For example, ‘good’ equals [0.8, 1], ‘bad’ equals [0, 0.2], etc. This method will not influ-ence the final result because it is a comparative evaluation.Taking delivery service, for example, there are three logistic companies for selection. Company

A (LA) has the shortest delivery time, but company C (LC) has the lowest delivery cost. In addition,LA and LB provide door-to-door conveyance, online search and other services, whereas LC doesnot provide such services; hence, the reliability of LA and LB is higher than LC.Then LA, LB and LC construct the plan layer corresponding to D2, D5 and D6, as shown in

Figure 5.Suppose that the delivery time of general logistic company is from 2 to 168 h, the delivery cost is

from 2 to 15 yuan and the reliability is denoted as a number between 0 and 1. The related indicatorsand normalized results are listed in Table VI.In terms of the weight of D2, D5 and D6, the synthesized weights of LA, LB and LC are

calculated as follows:

WLA = 0.867× 0.413 + 0.385× 0.03 + 0.9× 0.074= 0.436

WLB = 0.723× 0.413 + 0.538× 0.03 + 0.9× 0.074= 0.381

WLC = 0.434× 0.413 + 0.769× 0.03 + 0.3× 0.074= 0.225



Figure 5. Relationship between logistic companies and the indicators in SCL.

Table VI. Normalized indicators of LA, LB and LC.

Delivery Normalized Delivery Normalized Reliability oftime (h) delivery time cost (yuan) delivery cost delivery

LA 24 0.867 10 0.385 0.9LB 48 0.723 8 0.538 0.9LC 96 0.434 5 0.769 0.3

According to the above computation result, we can see that logistic company A provides moresatisfying service. This result agrees to the fact and also proves the soundness and the feasibilityof the proposed model and method.

6. CONCLUSIONS AND FUTURE WORK

Service-oriented business process shows some new characteristics in terms of execution mechanismand performance evaluation. In this paper, the definition and characteristics of SOBP are proposedwith a demonstrative scenario. In order to merge the business system and IT system, a service-oriented business performance evaluation model is depicted, based on which the KPIs in each layerare discussed, especially process-related indicators and activity-related indicators. The relationshipbetween business performance and IT performance is analyzed first from a qualitative point ofview through resolving a practical problem of an enterprise information system, and then froma quantitative point of view through calculating the correlation of KPIs in different layers. Animproved AHP method is presented for service selection, and the analysis of motivating exampleproves the correctness and the feasibility of our method.However, AHP has its own defects, such as the hard satisfaction of consistency of judgment matrix

and the subjectivity of the scoring by experts. Future efforts should be dedicated to improving AHPto make it more suitable to the application of business process performance evaluation.

ACKNOWLEDGEMENTS

The research of the paper is supported by the National Natural Science Foundation of China (grants 60674080and 60504030), the National High-Tech R&D (863) Plan of China (project no. 2006AA04Z166), the NationalBasic Research Program of China (project no. 2006CB705407) and the European Commission FP6 projectImportNET (project no. FP6-033610).



REFERENCES

1. Foster I, Kesselman C. The Grid: Blueprint for a new computing infrasture (2nd edn). China Machine Press: Beijing,2005.

2. Huhns MN, Singh MP. Service-oriented computing: Key concepts and principles. IEEE Internet Computing 2005;9(1):75–81.

3. Loo AW. The future of peer-to-peer computing. Communications of the ACM 2003; 46(9):56–61.4. Leymann F, Roller D, Schmidt MT. Web services and business process management. IBM Systems Journal 2002;

41(2):198–211.5. Andrews T, Curbera F, Dholakia H, Goland Y, Klein J, Leymann F, Liu K, Roller D, Smith D, Thatte S, Trickovic I,

Weerawarana S. Business Process Execution Language for Web Services, Version 1.1, May, 2003. http://www-128.ibm.com/developerworks/library/specification/ws-bpel/ [17 January 2007].

6. Jablonski S. Processes, Workflows, Web Service Flows: A Reconstruction (Lecture Notes in Computer Science, vol.3551). Springer: Berlin, 2005; 201–213.

7. Liu B, Cai SQ, Zheng SY. Evaluation index system for business processes. Journal of Huazhong University of Scienceand Technology 2005; 33(4):112–114.

8. Zhu JR, Liu DC, Tong W, Zheng L. Research on process-based manufacturing performance measurement system.Computer Integrated Manufacturing Systems 2005; 11(3):438–445.

9. Chen X. Performances analysis of workflows based on generalized stochastic petri nets. Computer IntegratedManufacturing Systems 2003; 9(5):399–402.

10. Spooner D, Cao J, Jarvis SA, He L, Nudd GR. Performance-aware workflow management for Grid computing. TheComputer Journal 2005; 48(3):347–357.

11. Menasce DA. QoS issues in Web services. IEEE Internet Computing 2002; 6(6):72–75.12. Song HG, Lee K. sPAC (Web Services Performance Analysis Center): Performance Analysis and Estimation Tool of Web

Services (Lecture Notes in Computer Science, vol. 3649). Springer: Berlin, 2005; 109–119.13. Chen J, Yang Y. A taxonomy of Grid workflow verification and validation. Concurrency and Computation:

Practice and Experience 2008. DOI: 10.1002/cpe.1220. Available at http://www.ict.swin.edu.au/personal/jchen/papers/VVTaxonomy.pdf.

14. Liu Y, Zhang ZG. Research on tasks ranking in workflow based on fuzzy analytic hierarchy process. Computer IntegratedManufacturing Systems 2006; 12(5):688–691.

15. Xie XQ, Chen KY. An AHP-based evaluation model for service composition. ICCSA 2006 (Lecture Notes in ComputerScience, vol. 3983). Springer: Berlin, 2006; 756–766.

16. Liu C, Shan ZG, Lin FY. Quality of Service of Computer Networks. Tshinghua University Press: Beijing, 2004.17. Zeng L, Benatallah B, Ngu AHH, Dumas M, Kalagnanam J, Chang H. QoS-aware middleware for web services

composition. IEEE Transactions on Software Engineering 2004; 30(5):311–327.18. Cardoso J, Sheth A, Miller J, Arnold J, Kochut K. Quality of service for workflows and web service processes. Journal

of Web Semantics 2004; 1(3):281–308.19. Saaty TL. The Analytical Hierarchical Process. McGraw-Hill: New York, 1980.


Documents

A service-oriented business performance evaluation model and the performance-aware service selection method