A holistic evaluation of the e-procurement website by using a hybrid MCDM methodology

Electronic Government, An International Journal, Vol. 10, No. 2, 2013 125

Copyright © 2013 Inderscience Enterprises Ltd.

A holistic evaluation of the e-procurement website by using a hybrid MCDM methodology

Mehmet Kabak* Department of Industrial and System Engineering, Turkish Military Academy (Harbiye), 06654 Bakanliklar, Ankara, Turkey E-mail: [email protected] *Corresponding author

Serhat Burmaoğlu Department of Business Administration, Turkish Military Academy (Harbiye), 06654 Bakanliklar, Ankara, Turkey E-mail: [email protected]

Abstract: Governments strive to enhance electronic applications in an attempt to take advantage of the benefits of the internet. The most important application of the e-government applications is electronic Public Procurement (e-PP) and it can be defined as the purchasing of goods and services via internet. Like other developed and developing countries, Turkey also shifted the public procurement system through electronic means. Therefore, this study is aimed to develop an evaluation model for web interface of the e-PP which is called EKAP. While developing the model, Decision Making Trial and Evaluation Laboratory (DEMATEL), Analytic Network Process (ANP) and fuzzy set theory are used. As a result, standardisation, links, reliability and navigability are found as the most important urgent need for improvement.

Keywords: e-PP; electronic public procurement; public procurement; evaluation; e-government; website; DEMATEL; decision making trial and evaluation laboratory; ANP; analytic network process; WVA; weight variance analysis; fuzzy.

Reference to this paper should be made as follows: Kabak, M. and Burmaoğlu, S. (2013) ‘A holistic evaluation of the e-procurement website by using a hybrid MCDM methodology’, Electronic Government, An International Journal, Vol. 10, No. 2, pp.125–150.

Biographical notes: Mehmet Kabak is a Lecturer in the Turkish Military Academy, Department of Industrial and System Engineering. He received his BS from the Turkish Military Academy, Systems Engineering in 1996. He has graduated from the School of Science Engineering and Technology, İstanbul Technical University in 2001. He has received his PhD from Operations Research (Marmara University) in 2006. His research interest is multi criteria decision making, fuzzy systems, operations research, energy, peace keeping operations, optimisation and modelling.

126 M. Kabak and S. Burmaoğlu

Serhat Burmaoğlu is an Assistant Professor at the Turkish Military Academy Leadership R&D Center. He received his BS from the Turkish Military Academy, Systems Engineering in 1997. He has graduated from the Defense Sciences Institute Logistics Management MBA Program in 2006. He has received his PhD from Quantitative Methods for Business Program (Atatürk University) in 2009. His research interest is investigating the relationship between economic growth, productivity, competitiveness, innovation and knowledge economy in macro-economic level by using multivariate statistical analysis and data mining applications for extracting usable patterns to direct development policies.

1 Introduction

E-government (e-government) becomes popular by the advancements on Information and Communication Technologies (ICTs). Increasing computer usage makes available for all stakeholders to engage some governmental procedures electronically in seconds. This makes e-government inter-related in nature.

Definition of e-government is introduced by Means and Schneider (2000) as relationships between governments and customers or suppliers by the use of electronic means. By the way, it can be understood from the definition that the most important aspect of e-government is the electronic linkage between government and other counterparts. In addition, Brown and Brudney (2001) categorised this linkage as Government-to-Government (G2G), Government-to-Citizen (G2C) and Government-to-Business (G2B). This categorisation may be extended to some other interactions, which have been materialised electronically.

According to Kaliannan et al. (2010), the most popular use of electronic means is e-Public Procurement (e-PP). e-PP is accepted as a general application of e-government and e-PP can be identified as a critical factor for the functioning of e-government. Kaliannan et al. (2010) defines electronic procurement (e-procurement) as “the use of electronic methods in every stage of the purchasing process from identification of requirements through payment, and potentially to contract management”. A number of public sector agencies worldwide have identified e-procurement as a priority of e-government agenda and have implemented or are in the process of implementing buy-side e-procurement systems (Croom and Brandon-Jones, 2004).

In addition to its priority, governments can economise by using e-PP tools. The savings are mostly thought that those are derived from administrative costs. On the other hand, several studies show that the most significant percentage of savings obtained from e-procurement arises from better sourcing decisions and not from reduced administrative costs (Baker, 1999).

Besides e-procurement’s benefits, the website design features are also important for diffusion of these applications. Because of its innovative nature, it is known that electronic applications mainly destroy traditional approaches. The visual appearance and user-friendly interface/software may facilitate the adaptation and diffusion process of these systems.

Hence, the aim of this study is to develop an evaluation model for the web interface of the e-PP platform of Turkey and apply the developed model to a real case for illustration. The e-PP platform will be evaluated by using three different approaches

A holistic evaluation of the e-procurement website 127

because the e-PP platform has three stakeholders (public authorities, suppliers and citizens). This triadic approach was mentioned by Suri and Sushil (2011) as the perspectives of stakeholders are not used commonly while measuring the performance of e-governance. In this study by using these perspectives, it is thought to overcome this shortcoming. Since multidimensional characteristics of website design require Multi-Criteria Decision Making (MCDM) methods in the evaluation process, a hybrid model based on Decision Making Trial and Evaluation Laboratory (DEMATEL) that is used for determination of relationship between criteria, Analytic Network Process (ANP), which is used for determination of criteria’s weights, and Weight Variance Analysis (WVA), which is used for sensitivity analysis, is proposed for the evaluation of website features of public procurement platform.

The rest of the paper is organised as follows: First, electronic Public Procurement (e-PP) literature is reviewed and Turkish Public Procurement Authority and e-procurement procedures are defined. In Section 3, the website evaluation literature is briefly explained and the variables which will be used for evaluation are determined with the aid of experts’ opinions. The details of the methods and the proposed hybrid model are explained and a real case application is executed in Sections 4 and 5, respectively. Finally, WVA is performed in Section 6 and conclusions are discussed.

2 Electronic Public Procurement and Turkey

E-procurement has been widely adopted by companies seeking better business processes and an improved bottom line. These advantages directed public authorities to think of the usage of electronic means for governments which engage in extensive buying activities and are major customers for a wide range of goods and services.

As it is known, public procurement can be defined as the purchasing of goods, services and construction works by government agencies and public authorities. So, public procurement is an important function of government (Thai, 2001). Because, it has to satisfy requirements for goods, works, systems and services in a timely manner. In addition, from the good governance point of view, it has to meet the basic principles as transparency, accountability and integrity (Wittig, 2003; Callender and Schapper, 2003). As a result of building an ideal e-PP website, these conditions will be more satisfied by participants. By the way, e-procurement converts traditional manual procurement processes in the government machinery to e-procurement on the internet. It uses internet technologies to bring government agencies in the country and suppliers around the world together into a virtual trading environment.

Because of the wide variety of applications and international trading nature of e-procurement, the Organization of Economic Cooperation and Development (OECD) puts forward some rules to regulate the e-procurement process for member countries. The OECD has identified ten principles in order to provide policy makers with principles for enhancing integrity throughout the entire public procurement cycle, taking into account international laws, as well as national laws and organisational structures of member countries. These include elements of transparency, good management, prevention of misconduct as well as accountability and control (Principles for Integrity in Public Procurement, 2009). Moreover, the European Union declared consolidated directive (2004/18/EC) to give a supportive policy approach to e-procurement adoption across the member states (EUROPA, 2004). Not only are the OECD regulations but also are the EU


directives important subjects for Turkey, because it is known that Turkey is a member of OECD and a candidate country of the EU.

Introducing e-procurement and automating the procurement process or parts of it has great potential in terms of accountability, transparency and cost savings (Schoenherr and Tummala, 2007; Yadav and Yadav, 2009). This in hand will determine quality in the product or service being received by the government agencies. E-procurement is an existing proof that the government is endeavouring in improving the public service delivery and a model of transparency and accountability. Although the service is upgraded and performance delivered, the people shall benefit lucratively and develop into a more robust nation. From another perspective, governments procure goods and use complex contractual systems designed to protect the public interest in order to preserve accountability and transparency services (Rasheed, 2004). The transparency and accountability issues are the most significant subjects in public procurement and these attributes are paramount as in regards to good governance (Schapper et al., 2006). Schapper et al. (2006) suggest that the increasing demand for transparency, efficiency and effectiveness in service quality is an important issue in public sector management. So, participation of citizens in the procurement process is also an important issue for transparency (Kaliannan et al., 2009).

The e-procurement issues have been discussed in many ways through different dimensions of users – the private or corporate sector, business-related organisations or government sector. The idea of e-procurement arises in a cross-functional activity represented in a variety of journals. However, the important point of the discussion is in emphasising the impact of information technology and systems on procurement practice and deals with emerging issues in the field (Schoenherr and Tummala, 2007). However, according to Thai (2001), public procurement has been a neglected area of academic education and research, although governmental entities, policy-makers and public procurement professionals have paid a great deal of attention to procurement improvements and reforms. On the other hand, Moe (2010) suggests that there are a lot of studies concerning procurement policies and procurement as an instrument to achieve specific goals in his literature review study. His literature review shows that there is no conducted study regarding e-procurement website quality.

Although there are various forms of e-procurement which concentrate on one or many stages of the procurement process such as e-tendering, e-marketplace, e-auction/ reverse auction and e-catalogue/purchasing, e-procurement can be viewed more broadly as an end-to-end solution that integrates and streamlines many procurement processes throughout the organisation. Although the term “end-to-end e-procurement” is popular, industry and academic analysts indicate that this ideal model is rarely achieved and e-procurement implementations generally involve a mixture of different models.

Finally, from these reviewed studies it can be understood that e-PP implementation in Turkey, as a developing country and member of OECD, can be thought as a requirement. Because of these aspects, requirements and benefits, Turkey decided to implement an e-PP platform. Before explaining the e-PP platform, the legal procurement system of Turkey is explained in the next sub-section.

2.1 E-procurement in Turkey

The first legal framework of public procurement system in the Republic of Turkey was the “Law on Purchasing and Selling Activities Carried Out on Behalf of Government”


enacted in 1925. Then, in 1934 “Law on Auctions, Reverse Auctions and Tendering” entered into effect for quite some time until 1983. In 1983, the “State Tender Law” came into force and regulated the public procurements prior to the comprehensive reforms in the area. For the purpose of ensuring harmonisation with international standards and EU norms, “Public Procurement Law” and “Public Procurement Contracts Law” were adopted by the parliament on 4 January 2002 and both laws came into effect as of 1 January 2003. Public procurement process has been regulated and adopted by these laws to international principles. The scope of the public procurement law is as wide as to cover procurements, which amounts to 10% of GDP, by all kinds of public entities and public economic enterprises as well as their specified partnership governed by public law, or under public control or using public funds (Republic of Turkey Public Procurement Authority, 2009).

One of the novelties of the law was establishment of an independent regulatory and monitoring body namely Public Procurement Authority. By doing so public funds can be used and controlled centrally. One of the main duties of the Public Procurement Authority is to establish and operate the e-PP platform up to the European Community Public Procurement Directives. For this purpose, the reform efforts have been enforced and e-signature law (No.: 5070 of 2004) and some arrangements in Public Procurement Law have been done by public authorities (Electronic Signature Law, 2010).

E-procurement model created by Public Procurement Authority aims to transform all the tender stages into electronic system, from need assessment by contracting entities to contract management process (EKAP, 2012). The E-procurement model includes all the stages, which are independent from each other. The components of the e-PP system are public authorities, suppliers and citizens. Although public authorities and suppliers have authorisation to access for preparing documents, citizens have the right to trace the process for transparency. Therefore, government authorities warehouse all data, which is materialising in stages for statistics and performance evaluation of both parties.

The first interaction of both administrations and suppliers is the web interface of the platform designed by Public Procurement Authority. Thus, visual impact of website and user-friendly features of software can be seen as an important issue for stakeholders. The study concentrates on this issue to determine which of the features are more important for efficient usage of e-procurement website in first touch. By exploring the dominant criteria, some strategic implications can be made for website designers to enhance diffusion of the e-procurement system on a maximum level.

3 Evaluation criteria for e-PP website

Due to the multidimensional characteristics of website design, it must be appropriately addressed by MCDM methods. Nowadays, there is no standardised model for evaluating websites, and the existing methods do not offer enough insights for e-procurement’s proprietors to determine whether its website meets the best practices from the perspectives of all stakeholders (Tsai et al., 2011).

An in-depth literature survey is conducted to determine the evaluation criteria of websites in general. According to Fassnacht and Koese (2006), Zeithaml, Parasuraman and Malhotra were the first authors who provided a formal definition of quality of electronic services: “the extent to which a website facilitates efficient and effective shopping, purchasing and delivery”. Since websites are not only means of shopping,


this definition can be accepted too narrow nowadays. The modification of Parasuraman et al.’s (2005) model comes up with the Kim et al.’s (2006) study in which E-S-QUAL was proposed for capturing service attributes of apparel retail websites. In the last decade many studies were conducted regarding website measurement. Tsai et al. (2010), Kaisara and Pather (2011), and Büyüközkan and Çiftçi’s (2012) studies are current and good examples for website measurement. However, it is known that reviewing the literature is not sufficient for defining the criteria. Because of the special aspects of study field, the criteria should be examined by experts also. There are 15 experts for determining the criteria. Four experts from public authorities were selected for reviewing the attributes and they have a minimum ten years experience on public procurement and two years experience on e-PP. In addition, they contributed voluntarily to this study. These experts were selected from the three different public institutions. Because of their restrictions, expert’s names and their institutions were not revealed in this study. In addition to experts from the public authorities, seven suppliers using e-PP platform frequently and four citizens who are aware of the procurement platform also contributed to the evaluation process. As a result of literature review and 15 experts’ direction, it was decided that ten criteria be used for the evaluation of administration and supplier interfaces of Public Procurement Platform of Turkey. Criteria and brief definitions are demonstrated in Table 1.

Table 1 Website’s evaluation criteria

Evaluation criterion Abb. Definition

Navigability C1 Easy to use and navigate in the site. How easy it is return to the homepage of the site, how easy it is to find relevant information, how many links are required to get from one point in a site to another and what search tools the site provides

Speed C2 Website downloads speed and easy and faster ways to access links Standardisation C3 Clearness, consistency and relevancy of the information content Links C4 Availability of the links to legal documents or forms concerning the

procedure of procurement law Accuracy C5 Up-to-date information. Therefore, the procurement law is

continuously changing in nature; site should give exact and correct information to parties

Richness C6 Detailed level and scope of information content. Site should give the users complete information about the usage of the system and give complete response for sustainable applications

Attractiveness C7 Websites’ exciting and enjoyable features. Website should attract the users via graphical shows. The website also should be designed visually

Reliability C8 This criterion deals with how a website proves to be trustworthy for citizens. A confident website should assure the secrecy of its users’ personal, private and firm data

Personalisation C9 This criterion includes an individualised interface. Customised content of the website can provide a user with the relevant and up-to-date information that will address his specific needs

Responsiveness C10 This criterion deals with the provision of information on Frequently Asked Questions (FAQs) and prompts assistance for solving problems


The e-PP system is a newly organised system in Turkey and it is really hard to compare these criteria without a reference value. While thinking about this limitation, some of these criteria seem not to be operationalised. Because there is a high level uncertainty, these criteria depend mostly on the judgements and interpretations of the experts. However, correspondingly, the aim of this study was to propose an evaluation model for e-PP of Turkey and apply the model with a real case application for illustration. Therefore, this research is conducted for the first time in Turkey besides its defined assumptions. For achieving this aim, a combined/hybrid model that consists of the DEMATEL method and ANP method is used to determine relationships between criteria and to compute weights of criteria and website’s satisfaction grades. In addition to overcoming the uncertainty problem, the fuzzy set theory is integrated in dealing with the vagueness of human thought and expression in decision making. Finally, WVA is used for sensitivity analysis.

4 Evaluation methods and the proposed model

In this section, some essentials of the fuzzy DEMATEL, fuzzy ANP and WVA are described as follows.

4.1 The FDEMATEL method

All criteria of the systems are mutually related in an interdependent system, directly or indirectly. As it affects all the others, it is difficult to find priorities for action. The science and human affairs program of the Battelle Memorial Institute of Geneva developed the DEMATEL method between 1972 and 1976. It converts the relationship between the causes and effects of criteria into an intelligible structural model of the system. It can also confirm the interdependence among the variables/attributes. It restricts the relation reflecting the characteristic with an essential system and development trend (Fontela and Gabus, 1974; Tsai et al., 2008; Li and Tzeng, 2009).

To visualise the structure of complicated causal relationships with matrices or digraphs, this method is considered as practical and useful. Digraphs are more useful than directionless graphs because of demonstrating the directed relationships of sub-systems. Also, the digraph portrays a basic concept of contextual relation among the elements of the system. In this system, the numeral represents the strength of influence. The DEMATEL is based on digraphs. It can separate involved criteria into cause group and effect group. For visualising the structure of complicated causal relationships with matrices or digraphs, the DEMATEL is accepted as useful and practical. Moreover, it has been applied in many situations, such as supplier selection, knowledge management, e-learning programmes, safety management systems, service quality, developing the competencies of global managers and companies (Liou et al., 2007; Mokhtarian, 2011; Lin et al., 2011; Chang et al., 2011).

Hence decision makers (DMs) generally fail to make a good numerical prediction for criteria; evaluation is expressed in linguistic terms. In addition, human judgement on qualitative attributes is always subjective and imprecise. Because of the imprecise expressions, the fuzzy set theory is commonly used in decision problems. Fuzzy linguistic models permit the translation of verbal expressions into numerical expressions. Fuzzy numbers expand on the idea of the confidence interval and are defined over a


fuzzy subset of real numbers. A Triangular Fuzzy Number (TFN) shown in Figure 1 is a type of fuzzy number and should possess the same basic properties.

Figure 1 A triangular fuzzy number, M

A fuzzy number M defined on ℜ is a TFN if its membership function [ ]( ) : 0,1M yµ ℜ → is equal to

( ) / ( ),( ) ( ) / ( ),

0, otherwiseM

y l m l l y my u y u m m y uµ

− − ≤ ≤= − − ≤ ≤

(1)

where l, u and m are real numbers and l ≤ m ≤ u. The linguistic variable scale and the corresponding TFNs used in this study are shown in Table 2 (Wu and Lee, 2007; Tooli et al., 2011; Kabak et al., 2012).

In this paper, the fuzzy set theory is incorporated with DEMATEL and ANP through an evaluation form that uses linguistic variables. The value of the linguistic variables that a DM has assigned to the pairwise comparison between each two criteria is converted into TFN scores. The FDEMATEL methodology which is applied in this paper is shown as follows (Chiu et al., 2006; Tsai and Chou, 2009; Jassbi et al., 2011):

Table 2 The fuzzy linguistic scale

Linguistic terms for importance

Linguistic terms for performance

TFN for DEMATEL TFN for ANP

TFN (reciprocal)

Equal important (E) Very poor (0, 0, 0.25) (1, 1, 1) (1, 1, 1) Weak important (W) Poor (0, 0.25, 0.50) (2, 3, 4) (1/4, 1/3, 1/2) Strong important (S) Fair (0.25, 0.50, 0.75) (4, 5, 6) (1/6, 1/5, 1/4) Demonstrated important (D) Good (0.50, 0.75, 1) (6, 7, 8) (1/8, 1/7, 1/6) Absolute important (A) Very good (0.75, 1, 1) (8, 9, 10) (1/10, 1/9, 1/8)

Step 1: Producing the fuzzy direct-relation matrix

Respondents are asked to indicate the degree of direct influence each factor/element i exerts on each factor/element j. If i is equal to j, the degree of direct influence is none

11 22( ... (0,0,0)).a a= = = By using the pairwise comparison scale shown in Table 1,


let ( , , )k k k kij ij ij ijx l m u= indicate the fuzzy assessments of DM k (k = 1, 2, … h) about

the degree to which the criterion i affects criterion j. The fuzzy average matrix A is then calculated, where each element ija is computed by averaging all the DMs’ TFN scores as:

1 1

1 1( , , ) ( , , ).h h

k k k kij ij ij ij ij ij ij ij

k k

a l m u x l m uh h= =

= = =∑ ∑ (2)

Then, fuzzy matrix A is produced which is shown as

1 1

1

1

0

0

j n

i ij in

n nj

a a

A a a a

a a

=

(3)

which is called initial direct-relation fuzzy matrix.

Step 2: Normalising the direct-relation fuzzy matrix

On the basis of the direct-relation fuzzy matrix ,A the normalised direct-relation fuzzy matrix M can be obtained by using equations (4) and (5) in which all principal diagonal elements are equal to zero.

M k A= ⋅ (4)

1 11 1

1 1Min , , {1,2,3,..., }.max max

n n

ij iji n j nj i

k i j nu u

≤ ≤ ≤ ≤= =

= ∈

∑ ∑ (5)

Step 3: Derive the total-relation fuzzy matrix

After the normalised direct-relation fuzzy matrix M has been obtained, the total-relation fuzzy matrix S can be derived by using equation (6), where I is denoted as the identity matrix.

2 3

1

1( ) .

i

İS M M M M

S M I M

∞

=

−

= + + + =

= −

∑ (6)

Step 4: Compute dispatcher group and receiver group

By producing matrix ,S then i iD R+ and i iD R− are calculated in which iD and iR are the sum of row and the sum of columns of ,S respectively. To finalise the procedure, all calculated i iD R+ and i iD R− are defuzzified through the suitable defuzzification method. The Centre of Area (COA) defuzzification method is used to determine the Best Non-fuzzy Performance (BNP) value of the fuzzy numbers mainly because it is practical (Onut et al., 2009). The BNP value of a fuzzy number can be calculated as:


( ) ( )BNP .

3ij ij ij ij ij

ij

l u l m l + − + − = (7)

Step 5: Set threshold value and obtain the impact-digraph-map

If all the information from matrix S converts to the impact-digraph-map, the map would be too complex to show the necessary information for decision making. To obtain an appropriate impact-digraph-map, the decision maker must set a threshold value for the influence level. Only some elements, whose influence levels are higher than the threshold value, can be chosen and converted into the impact-digraph-map. The threshold value can be decided through the brainstorming of the decision makers or experts. An impact-digraph-map can be acquired by mapping the dataset of ( , ),D R D R+ − where the horizontal axis represents ( )D R+ and the vertical axis represents ( ).D R− Using the values of ( )i iD R+ and ( ),i iD R− a level of criterion’s influence to others and a level of relationship with others are defined. If ( )i iD R− is positive, then the criterion is affecting other criteria and is assumed to have higher priority and is called dispatcher. Any criterion having negative values of ( )i iD R− receiving more influence from another is assumed to have a lower priority and is called receiver. On the other hand, ( )i iD R+ provides an index of the strength of influences given and received, that is, ( )i iD R+ shows the degree of the central role that criterion i plays in the problem (Seyed-Hosseini et al., 2005; Tzeng et al., 2007; Wu and Lee, 2007).

4.2 The FANP method

The Analytic Network Process (ANP) allows for complex interrelationships among decision levels and attributes (Saaty, 1996). The ANP feedback approach replaces hierarchies (Figure 2(a)) with networks (Figure 2(b)) in which the relationships between levels cannot be easily represented as higher or lower, dominant or subordinate, direct or indirect (Dağdeviren, 2010).

Figure 2 Hierarchy and network: (a) hierarchy and (b) network

(a) (b)


In ANP, the modelling process can be divided into three steps, which are described as follows (Dağdeviren, 2010):

Step 1: The pairwise comparisons and relative weight estimation

Before performing the pairwise comparisons, all criteria and clusters compared are linked to each other. The pairwise comparisons are made depending on the scale shown in Table 1. In the pairwise comparison matrix, the score of aij represents the relative importance of the component on row (i) over the component on column (j), i.e., / .ij i ja w w= The reciprocal value of the expression (1/ )ija is used when the component j is more important than the component i. The comparison matrix A is defined as

1 1 1 2 1 12 1

2 1 2 2 2 12 2

1 2 1 2

/ / / 1/ / / 1/ 1

.

/ / / 1/ 1/ 1

n n

n n

n n n n n n

w w w w w w a aw w w w w w a a

A

w w w w w w a a

= =

(8)

Then, a local priority vector (eigenvector) w is computed as an estimate of the relative importance accompanied by the elements being compared by solving the following equation:

max ,Aw wλ= (9)

where maxλ is the largest eigenvalue of matrix A.

Step 2: Formation of the initial supermatrix

The obtained vectors are further normalised to represent the local weight vector. Supermatix is formed, local weight vectors are entered in the appropriate columns of the matrix of influence among the elements, to obtain global priorities. The supermatrix representation of a network with three levels is given as follows (Figure 2(b)):

21 22

32

Goal( )0 0 0

Criteria( ) ,0

Alternatives( )0

G C AG

W CW W

AW I

=

(10)

where W21 is a vector that represents the impact of the goal on the criteria, W22 is a vector that represents impact of the interdependences among criteria, W32 is also a vector that represents the impact of criteria on each of alternatives and I is the identity matrix. Any zero value in the super-matrix can be replaced by a matrix if there is an interrelationship of elements within a cluster or between the clusters.

Step 3: Formation of the weighted super-matrix

An eigenvector is obtained from the pair-wise comparison matrix of the row clusters with respect to the column cluster, which in turn yields an eigenvector for each column cluster. The first entry of the respective eigenvector for each column cluster is multiplied by all the elements in the first cluster of that column, the second by all the elements in the


second cluster of that column and so on. In this way, the cluster in each column of the super-matrix is weighted, and the result, known as the weighted super-matrix, is stochastic (Ramik, 2006).

DM cannot always explain his judgements about certain attributes, quality, performance, etc., with discrete scales. For these reasons fuzzy scales are defined. In the application, TFNs have been used by DM to state their preferences to compare attributes.

In the proposed methodology, pair-wise comparison matrices are formed with the help of TFNs, the FANP has been used to determine weights of personnel selection criteria. The FANP can easily accommodate the interrelationships existing among the functional activities. The concept of supermatrices is employed to obtain the composite weights that overcome the existing interrelationships.

Pairwise comparison matrices are structured by using TFNs (l, m, u). The fuzzy matrix can be given as follows:

11 11 11 12 12 12 1 1 1

21 21 21 22 22 22 2 2 2

1 1 1 1 1 1

( , , ) ( , , ) ... ( , , )

( , , ) ( , , ) ... ( , , ).

( , , ) ( , , ) ... ( , , )

l m u l m u l m un n n

l m u l m u l m un n n

l m u l m u l m um m m m m m mn mn mn

a a a a a a a a a

a a a a a a a a aA

a a a a a a a a a

=

(11)

The amn represents the of comparison m (row) with component n (column). The pair-wise comparison matrix ( )A is assumed as reciprocal.

12 12 12 1 1 1

2 2 212 12 12

1 1 1 2 2 2

(1,1,1) ( , , ) ... ( , , )

1 1 1, , (1,1,1) ... ( , , ).

1 1 1 1 1 1, , , , ... (1,1,1)

l m u l m un n n

l m un n nu m l

u m l u m ln n n n n n

a a a a a a

a a aa a a

A

a a a a a a

=

(12)

In this study, the logarithmic least squares method is used for getting estimates for fuzzy priorities iw . The logarithmic least squares method for calculating triangular fuzzy weights can be given as follows (Ramik, 2006):

( ), , 1, 2,3,..., ,l m uk k kW W W W k n= = (13)

where

( )( )

1/

1

1/

1 1

, { , , }.

nn skjjs

k nnn miji j

aW s l m u

a

=

= =

= ∈∏

∑ ∏ (14)

The experts evaluate alternatives by using linguistic scales shown in Table 2. For pairwise comparisons, rating that is preferred on the situation of high number of criteria and candidates or other ranking methods, i.e., TOPSIS, PROMETHEE, could be used to evaluate the performance of the alternatives. In this paper rating is used, overall


priorities for alternatives and crisp values are calculated by using matrix operations. Then, the alternative with the largest priority should be selected.

4.3 The WVA method

The WVA, which is based on the concept of the Importance Performance Analysis (IPA) (Martilla and James, 1977), suggests improvement actions. However, some researchers Tsai et al. (2010) substituted the performance variance rate and the ANP weights with the performance component and the importance, respectively, as is done in this study.

WVA provides a more detailed analysis focusing on the gaps between an actual solution and an ideal solution according to particular evaluation criteria. In the proposed model, the performance variance rate is plotted against the ANP weight to provide a graphic illustration of which evaluation criteria are most in need of enhancement. The graphic in which the “ANP weight” constitutes the vertical axis and the “performance variance rate” constitutes the horizontal axis is labelled as the weight-variance map. This map is divided into four zones, as shown in Figure 3 (Tsai et al., 2011).

Zone 1 (Concentrate here): Criteria in Zone 1 are rated as having high weight and a high performance variance rate. Criteria in this zone are in most urgent need of improvement.

Zone 2 (Keep up the good work): Criteria located in Zone 2 are deemed important to evaluators and their performance variance rates are low. DMs need to carefully monitor these criteria to ensure that low performance variance rate levels are continued.

Zone 3 (Redeploy resources): Criteria falling in this zone have low importance and the performance variance rate. DMs should not be overly concerned about these criteria. Resources currently committed to improvement in Zone 3 could be reallocated to potentially more effective utilisations in Zone 1.

Zone 4 (Low priority): Criteria in Zone 4 are rated as having low importance and a high performance variance rate. As a result, it is not necessary to focus additional effort or resources to criteria in this zone.

4.4 The proposed model

The Turkish government has started e-government application to take the advantage of internet parallel to other developed countries. A web-based platform has been built and the government has declared that all public organisations have to use it to procure goods, services, etc. The procurement platform is mainly used by public authorities and suppliers and it can also be monitored by citizens as a right to trace the governmental issues for transparency. The website’s designers normally try to modify and revise its aspects in order to give a better service according to the stakeholder’s opinions. In this study, a hybrid model based on public authorities’, suppliers’ and citizens’ ideas is proposed to the web designers to modify the website as shown in Figure 3.

As shown in Figure 1, the fuzzy set theory is incorporated with MCDM methods, DEMATEL and ANP in the evaluation process of the e-PP website. Many MCDM methods of which the most popular one is the Analytic Hierarchy Process (AHP) are based on the concept of independence assumption between criteria, but this is not always true. The ANP feedback approach provides networks in which the relationships between levels cannot be easily represented as higher or lower, dominant or subordinate, direct or


indirect. It allows for complex interrelationships among participants (Saaty and Vargas, 1998; Meade and Sarkis, 1999). Hence, the ANP is applied to determine the evaluation criteria weights in this study. On the other hand, the treatments of inner dependences can theoretically use the ANP, but a more wise option is to employ DEMATEL. Before forming matrix calculations in ANP, the treatment of inner dependences needs to employ DEMATEL. It can be used to handle the inner dependences within a set of criteria. In addition, it can convert the relationships between cause and effect of criteria into a visual structural model (Fontela and Gabus, 1974; Tzeng et al., 2007; Shieh et al., 2010; Chen and Sun, 2012).

Figure 3 Framework of the proposed model

In this study, the website evaluation criteria, criteria weights and scores are determined; a detailed analysis is executed by following the FDEMATEL and FANP methods with a six-step procedure as shown in Figure 1. The WVA which focuses on the gaps between


an actual website and an ideal website according to particular evaluation criteria is applied.

5 A real case application

Public sector, which is normally on non-profit basis, has generally failed to keep up with the private sector in the pursuit of service excellence. In spite of that, the quality ideology became widespread in the public services with the total quality management movement in the early 1990s, and thus many governments nowadays try to meet the service expectations of their citizens (Kaisara and Pather, 2011). One of the most important tools to satisfy citizens’ expectations is the internet. Governments could use it to diminish cost and time in process and to give satisfactory service to citizens. Since the mid-1990s, use of the internet in Turkey has shown rapid development. The internet in Turkey has touched almost all sectors, including banking, education, health, transportation, etc. As mentioned before, the Turkish government also has ordered all governmental organisations to use e-PP’s website for procurement.

As a result of building an ideal e-PP website, the main characteristics such as transparency, accountability and integrity will be more satisfied by participants. In this paper, a hybrid MCDM tool is proposed for achieving this aim. This paper focuses on evaluating the e-PP website from three perspectives: public authorities, suppliers and citizens. In the application, first the experts, taking a part in e-PP website evaluation process, are determined. Four of the 15 experts are public authorities who use the website to procure foods and transportation service for their organisation. Seven of the experts are representatives of firms which supply goods or services to governmental organisations. Others are citizens, two of them are academic personnel having published papers on procurement and the last two are authors of this paper. This study uses ten criteria shown in Table 1 to evaluate the website. The FDEMATEL method is used to evaluate the influence of each criterion in website evaluation. A questionnaire for FDEMATEL and FANP composed of three parts is designed first in this research. The first part of the questionnaire outlines each criteria definition for easy understanding and response. Then, respondents are asked to compare the importance of each criterion using scales shown in Table 2. After the FDEMATEL calculations, the threshold values are determined and relationships between criteria are determined and shown in Figure 4. The final part is composed with the pairwise comparisons between each two criteria according to the relationships.

The proposed model is applied in a six-step procedure. First, the hierarchical structure shown in Figure 3 for the proposed model is formed. In Step 2, the experts adopt the criteria and choose the linguistic scale for making assessments. In Step 3, the relationships between those criteria are measured by the experts. Every expert is asked to make pairwise relationships between each pair of criteria. After averaging all these assessment matrices, the experts will have initial-direct relation fuzzy matrix .A The obtained results are shown in Table 3.

In Step 4, based on the initial direct-relation fuzzy matrix, the normalised direct-relation fuzzy matrix M is obtained. Next, the total-relation fuzzy matrix S and crisp values are calculated and the total-relation matrix is depicted as shown in Table 5. To access the casual relationships between criteria, values of i iD R+ , i iD R− are


calculated and defuzzified through the COA method and values are depicted in Table 4. Positive values of crisp( )i iD R− shown in Table 4 indicate that related criteria belong to the dispatcher group whereas negative criteria belong to the receiver group. The criterion C1 has the highest relationship with others according to the values of ( ).i iD R− Moreover, the same conclusions could be seen in the impact-diagraph-map after it is acquired by mapping a dataset of ( , ).i i i iD R D R+ −

In this study, a threshold value (p) of 0.19 is decided on, in consultation with the experts. This number is the most appropriate value to acquire a suitable relationship from trying above and under this number. The value under 0.19 gains too many criteria and complex relationships in the whole system. Based on the above threshold value, the impact-diagraph-map is obtained as shown in Figure 4. It is clear that the website evaluation criteria are visually divided into the dispatcher group, including C3, C4, C5 and C9 whereas the receiver group is composed of such criteria as C1, C2, C6, C7, C8 and C10. Therefore, the impact-diagraph-map shows that C3, C4, C5 and C9 affect others.

Valuable cues can be obtained from the impact-diagraph-map as shown in Figure 4. If DMs want to obtain high performances in terms of the receiver group criteria, it would be necessary to control and pay a great deal of attention to the dispatcher group criteria beforehand. This is because the dispatcher group criteria are difficult to move, whereas the receiver group criteria are easily moved (Hori and Shimizu, 1999). Hence, among these ten criteria, criterion C1 has the highest intensity in relation to other criteria and could be easily affected by the others, but criterion C3 is the most influencing one and it is quite difficult to move.

The arrows shown in Figure 4 or bold values shown in Table 5 represent the interdependence among the criteria. The interdependencies among criteria and by this way the effects of the criteria on each other are analysed. The arrow from C4 to C9 means that ‘Links’ effects ‘Personalisation’. The arrow between C1 and C10 means that they affect each other.

Figure 4 The impact-diagraph-map


Table 3 The initial direct-relation fuzzy matrix

C1 C2 C3 C4 C5

C1 (0, 0, 0) (0.54, 0.79, 0.96) (0.21, 0.43, 0.64) (0.21, 0.46, 0.71) (0.14, 0.21, 0.43) C2 (0.75, 1.00, 1.00) (0, 0, 0) (0.29, 0.50, 0.71) (0.32, 0.54, 0.75) (0.11, 0.21, 0.43) C3 (0.5, 0.75, 0.96) (0.25, 0.43, 0.68) (0, 0, 0) (0.36, 0.57, 0.79) (0.43, 0.68, 0.82) C4 (0.46, 0.68, 0.86) (0.39, 0.61, 0.82) (0.32, 0.57, 0.79) (0, 0, 0) (0.46, 0.68, 0.86) C5 (0.25, 0.46, 0.68) (0.21, 0.39, 0.64) (0.39, 0.61, 0.79) (0.36, 0.61, 0.79) (0, 0, 0) C6 (0.29, 0.50, 0.71) (0.14, 0.32, 0.57) (0.36, 0.57, 0.75) (0.29, 0.54, 0.75) (0.36, 0.61, 0.79) C7 (0.39, 0.61, 0.79) (0.21, 0.39, 0.64) (0.07, 0.29, 0.54) (0.14, 0.32, 0.57) (0.07, 0.18, 0.43) C8 (0.25, 0.43, 0.64) (0.21, 0.36, 0.57) (0.04, 0.14, 0.39) (0.04, 0.18, 0.43) (0.07, 0.18, 0.43) C9 (0.32, 0.57, 0.82) (0.18, 0.43, 0.68) (0.25, 0.39, 0.57) (0.04, 0.25, 0.54) (0.07, 0.25, 0.50) C10 (0.5, 0.71, 0.82) (0.46, 0.64, 0.79) (0.18, 0.32, 0.57) (0.25, 0.43, 0.68) (0.14, 0.29, 0.54)

C6 C7 C8 C9 C10 C1 (0.25, 0.39, 0.64) (0.36, 0.61, 0.82) (0.21, 0.39, 0.61) (0.29, 0.50, 0.75) (0.57, 0.79, 0.89) C2 (0.11, 0.29, 0.54) (0.14, 0.29, 0.54) (0.11, 0.25, 0.50) (0.21, 0.43, 0.68) (0.61, 0.86, 0.96) C3 (0.43, 0.64, 0.82) (0.5, 0.75, 0.93) (0.14, 0.36, 0.61) (0.18, 0.36, 0.61) (0.43, 0.68, 0.86) C4 (0.36, 0.61, 0.82) (0.36, 0.57, 0.79) (0.11, 0.32, 0.57) (0.11, 0.32, 0.57) (0.25, 0.50, 0.75) C5 (0.43, 0.68, 0.86) (0.36, 0.61, 0.79) (0.18, 0.36, 0.61) (0, 0.18, 0.43) (0.07, 0.29, 0.54) C6 (0, 0, 0) (0.32, 0.54, 0.71) (0.07, 0.21, 0.46) (0.04, 0.21, 0.46) (0.04, 0.21, 0.46) C7 (0.29, 0.50, 0.75) (0, 0, 0) (0.04, 0.14, 0.39) (0, 0.21, 0.46) (0.04, 0.18, 0.43) C8 (0.07, 0.25, 0.50) (0.07, 0.25, 0.50) (0, 0, 0) (0, 0.18, 0.43) (0.11, 0.25, 0.50) C9 (0.21, 0.46, 0.68) (0.36, 0.57, 0.79) (0.11, 0.25, 0.50) (0, 0, 0) (0.18, 0.39, 0.64) C10 (0.04, 0.18, 0.43) (0.18, 0.36, 0.57) (0.07, 0.21, 0.46) (0.11, 0.29, 0.54) (0, 0, 0)

Table 4 The fuzzy and crisp values of (Di + Ri), (Di – Ri)

Di Ri Di + Ri Di – Ri (Di + Ri)crisp (Di – Ri)crisp

C1 (0.53, 1.32, 4.52) (0.71, 1.66, 5.1) (1.24, 2.98, 9.62) (–0.18, –0.34, –0.58) 3.79 –0.25

C2 (0.52, 1.29, 4.34) (0.51, 1.3, 4.52) (1.03, 2.59, 8.86) (0.01, –0.01, –0.18) 3.47 –0.06

C3 (0.61, 1.5, 4.94) (0.41, 1.13, 4.12) (1.02, 2.63, 9.05) (0.21, 0.37, 0.82) 3.55 0.33

C4 (0.55, 1.42, 4.79) (0.4, 1.14, 4.27) (0.94, 2.56, 9.06) (0.15, 0.28, 0.52) 3.56 0.21

C5 (0.44, 1.23, 4.33) (0.36, 0.95, 3.74) (0.8, 2.18, 8.07) (0.08, 0.28, 0.58) 3.15 0.26

C6 (0.38, 1.11, 4.08) (0.42, 1.15, 4.24) (0.8, 2.26, 8.31) (–0.04, –0.04, –0.16) 3.26 –0.05

C7 (0.25, 0.86, 3.61) (0.51, 1.32, 4.56) (0.76, 2.18, 8.17) (–0.26, –0.46, –0.95) 3.19 –0.39

C8 (0.17, 0.66, 3.16) (0.2, 0.74, 3.33) (0.37, 1.4, 6.49) (–0.03, –0.08, –0.17) 2.51 –0.07

C9 (0.33, 1.05, 4.04) (0.19, 0.82, 3.52) (0.52, 1.87, 7.56) (0.14, 0.23, 0.53) 2.97 0.21

C10 (0.39, 1.03, 3.88) (0.46, 1.25, 4.29) (0.84, 2.28, 8.17) (–0.07, –0.22, –0.41) 3.20 –0.19

The weights of criteria to be used in the evaluation process are assigned by using the FANP method. In this phase, the experts are given the task of forming individual pairwise comparison matrix by using the scale given in Table 2. For this aim, first the pairwise comparison matrices are formed by the experts and local weights of the criteria are


calculated. For example, criterion 1 (C1) is compared with criterion (C2) using the questions “Which is considered more important by the expert evaluating the website” and “how much more important is it with respect to satisfaction with the website?” All the evaluation matrices are produced in the same manner. The criteria local weights (W21) are calculated.

In the last step, interdependent weights of criteria are calculated and the dependencies among the criteria are considered. Based on the dependencies calculated by FDEMATEL, the experts defined dependencies among all criteria via a pairwise comparison matrix. The results of relative importance weights of the matrix are calculated and listed in Table 6. The value ‘(0.00, 0.00, 0.25)’ presented in Table 6 means that there is no dependence between two criteria and the numerical values show the degree of relative impact between two criteria. For example, the C3’s degree of relative impact for C1 is (0.33, 0.58, 0.78).

Table 5 The total-relation matrix

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C1 0.18 0.20 0.19 0.19 0.15 0.18 0.20 0.14 0.19 0.19 C2 0.22 0.15 0.17 0.18 0.15 0.17 0.18 0.14 0.15 0.19 C3 0.24 0.21 0.16 0.20 0.19 0.20 0.22 0.19 0.16 0.21 C4 0.23 0.21 0.19 0.16 0.19 0.20 0.21 0.19 0.19 0.20 C5 0.21 0.18 0.19 0.18 0.12 0.18 0.19 0.14 0.14 0.17 C6 0.20 0.17 0.17 0.17 0.16 0.13 0.18 0.13 0.13 0.16 C7 0.18 0.16 0.14 0.15 0.12 0.15 0.12 0.11 0.12 0.14 C8 0.19 0.19 0.12 0.12 0.11 0.19 0.19 0.08 0.11 0.13 C9 0.20 0.18 0.16 0.16 0.14 0.17 0.18 0.13 0.11 0.17 C10 0.19 0.17 0.15 0.16 0.14 0.15 0.17 0.12 0.13 0.13

The bold values represent values higher than the threshold value (p = 0.19).

Global weights of the criteria on the basis of interdependence can be calculated by using the data given in Table 6 as follows:

cri 22 21 22

C1 (0.11,0.13,0.16)C2 (0.08,0.10,0.12)C3 (0.09, 0.11, 0.13)C4 (0.04, 0.05, 0.06)C5 (0.11, 0.13, 0.15)C6 (0.09, 0.10, 0.12)C7 (0.05, 0.07, 0.08)C8 (0.13, C9C10

w W W W

= = × = ×

(0.09,0.21,0.55)(0.05,0.11,0.43)(0.16,0.36,0.73)(0.16,0.38,0.75)(0.06,0.13,0.45)(0.04,0.08,0.38)(0.00,0.00,0.30)

0.17, 0.21) (0.00,0.00,0(0.05,0.06,0.08)(0.07,0.09,0.11)

=

.

.30)(0.02,0.05,0.36)(0.02,0.06,0.37)


Significant differences are observed in the results obtained for the criteria priorities when interdependent priorities of the criteria (wcriteria) and dependencies are not taken into account. For example, the result for C1 changes from (0.11, 0.13, 0.16) to (0.09, 0.21, 0.55). Crisp values are calculated using equation (7) and normalised. The same procedure for authorities’ and suppliers’ preferences is followed and criteria weights for all participants of e-procurement are found as shown in Table 7.

Table 6 Degree of relative impact for criteria (W22)

C1 C2 C3 C4 C5 C1 (0, 0, 0.25) (0.35, 0.6, 0.8) (0.18, 0.38, 0.63) (0.33, 0.58, 0.75) (0, 0, 0.25) C2 (0.35, 0.6, 0.8) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C3 (0.33, 0.58, 0.78) (0.23, 0.48, 0.73) (0, 0, 0.25) (0.2, 0.45, 0.7) (0.28, 0.45, 0.68) C4 (0.35, 0.55, 0.75) (0.2, 0.43, 0.68) (0.23, 0.48, 0.73) (0, 0, 0.25) (0.18, 0.38, 0.63) C5 (0.18, 0.33, 0.58) (0, 0, 0.25) (0.3, 0.5, 0.73) (0, 0, 0.25) (0, 0, 0.25) C6 (0.35, 0.58, 0.75) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C7 (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C8 (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C9 (0.18, 0.38, 0.6) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C10 (0.2, 0.43, 0.68) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C6 C7 C8 C9 C10 C1 (0, 0, 0.25) (0.23, 0.48, 0.73) (0, 0, 0.25) (0.18, 0.33, 0.55) (0.18, 0.38, 0.63) C2 (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0.18, 0.4, 0.65) C3 (0.28, 0.48, 0.73) (0.28, 0.53, 0.78) (0.1, 0.18, 0.43) (0, 0, 0.25) (0.3, 0.55, 0.8) C4 (0.2, 0.43, 0.65) (0.25, 0.48, 0.7) (0.13, 0.25, 0.48) (0.13, 0.28, 0.53) (0.18, 0.4, 0.65) C5 (0, 0, 0.25) (0.25, 0.45, 0.7) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C6 (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C7 (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C8 (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C9 (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) C10 (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25) (0, 0, 0.25)

Table 7 Criteria weights for participants

Criteria Citizens (C) Authorities (A) Suppliers (S) C1 0.12 0.11 0.11 C2 0.09 0.09 0.07 C3 0.17 0.16 0.18 C4 0.18 0.17 0.16 C5 0.10 0.09 0.11 C6 0.08 0.07 0.06 C7 0.06 0.05 0.04 C8 0.06 0.12 0.15 C9 0.07 0.07 0.06 C10 0.07 0.07 0.07


According to the results, the three most significant criteria for citizens are C4, C3 and C1, respectively. On the other hand, C4, C3 and C8 must take the first three places according to the authorities and suppliers. It is normal for citizens who are not related directly with the e-procurement not to concentrate on reliability. Under the perspective of e-procurement, the criteria C4, C3, C8 and C1 are key factors for successful web-based procurement.

6 The website’s performance evaluation

After determining the criteria weights, the stakeholders are separately asked to evaluate the website’s performance. The performance values, which are very good, good, median, poor, and very poor, are transformed by scaling them into the numbers 100, 75, 50, 25 and 0, respectively (Kim et al., 2006). The obtained evaluation result is shown in Table 8. Citizens give the highest global performance value (98.11) to C5 whereas suppliers give the lowest score (51.42) to C6. Finally, it is determined that a score of 100 represents the positive ideal solution *( )iX and a score of 0 represents the negative ideal solution .iX − The performance matrix is calculated by using the formula: * *( ) / ( )i ij i iX X X X −− − to obtain performance variance rates between the status quo and the ideal point for the website as shown in Table 8.

A coordinate diagram is acquired by mapping the dataset of Table 7 and three weight variance maps are illustrated for stakeholders. The map based on citizens’ evaluations is shown in Figure 5 and similar maps are illustrated for public authorities and suppliers. The maps could be further divided into four zones each by using the mean values at the crosshair points of the x- and y-axis (i.e., the overall average performance variance rate (0.267) and the average ANP weights (0.1) of the three different evaluations).

As seen in Figure 5 showing citizens’ evaluation, C1, C3 and C4 are located in Zone 1. Hence, these criteria are in the most urgent need of improvement. Citizens suggest that the e-procurement web design strategy should be adjusted based on ‘navigability (C1)’, ‘standardisation (C3)’ and ‘links (C4)’ for improving these weaknesses. When Zone 2 is examined, it can be seen that there is only one criterion located in Zone 2; no further improvement is required for this criterion. Managers must try to locate other criteria in this zone. Next, four criteria are identified in Zone 3 that are rated as having a low importance and a low variance rate. This implies that resources currently committed to improvement in Zone 3 could be reallocated to potentially more effective utilisations in Zone 1 without significant detriment to overall performance. Finally, two criteria falling into Zone 4 are a second priority for remedial action. Actions on these criteria should be delayed until the most improvement in Zone 1 is completed.

The navigability (C1) criterion is located in Zone 2 and Zone 1 based on the authorities’ and suppliers’ evaluation, respectively. Although authorities use a similar procedure for different executions, suppliers need more links for relevant information to be able to gain a competitive advantage. Suppliers also focus on the urgent development of navigability and web-designers could increase its usage of user-friendly features such as “set as home page”, “add to favourites” and “quick links”. High weights and high-performance variance rates appear for standardisation (C3) and links (C4) criteria based on citizens’ and authorities’ evaluation. The website should provide more standard


information and more related websites for these criteria. On the other hand, the current situation of C3 and C4 is satisfactory for suppliers.

Table 8 Performance scores for the website

C A S Average Website’s average performance scores according to the stakeholders C1 52.81 83.62 65.13 67.19 C2 83.22 54.15 52.46 63.28 C3 51.43 59.87 84.11 65.14 C4 54.13 56.02 85.24 65.13 C5 98.11 54.13 83.65 78.63 C6 97.52 76.19 51.42 75.04 C7 93.65 78.18 76.95 82.93 C8 54.57 78.14 94.89 75.87 C9 51.68 64.59 96.08 70.78 C10 75.67 98.11 92.52 88.77 Average performance scores 71.28 70.3 78.25 73.27

Performance variance rates for the e-PP’s website C1 0.47 0.16 0.35 0.328 C2 0.17 0.46 0.48 0.367 C3 0.49 0.40 0.16 0.349 C4 0.46 0.44 0.15 0.349 C5 0.02 0.46 0.16 0.214 C6 0.02 0.24 0.49 0.250 C7 0.06 0.22 0.23 0.171 C8 0.45 0.22 0.05 0.241 C9 0.48 0.35 0.04 0.292 C10 0.24 0.02 0.07 0.112 Overall average variance rate 0.287 0.297 0.218 0.267

Depending on the budget, authorities aim to have better performance values on C2, C5 and C9 whereas suppliers focus on C2 and C6. In addition, these criteria are located in low priority zones and managers should not be overly concerned with them. Actions on these criteria should be delayed until most of the improvement in the “Concentrate Here” zone has been completed. Connection speed to internet can be fastened and download time for contents must be shortened to support the speed (C2) criterion.

Criteria C6, C7, C10; C7, C9 and C10 are located in redeploy sources zone according to the evaluation of authorities’ and suppliers’ evaluation, respectively. Managers must consider the possibility of redeploying resources to remedial action in the concentrate here zone. Authorities are satisfied with criteria C1 and C8 and suppliers with C3, C4, C5 and C8. Contrarily, there is no criterion located in Zone 2 for citizens.


In this paper, a hybrid model by integrating DEMATEL, ANP and WVA is proposed that the designers of the e-PP website can see the evaluation of the website based on three stakeholders, and then undertake actions to achieve ideal quality levels.

Figure 5 The weight-variance map based on citizens’ evaluation

7 Conclusion

Governments start to use electronic applications by the advancements on ICTs. They intend to save cost and time in governmental procedures and to give chance to citizens to be able to monitor issues. For this aim, governmental organisations design websites for their own applications. These websites need to be modified to provide better services according to users’ expectations. This paper offers a hybrid model combining the FDEMATEL method, the FANP method and the WVA method for evaluating the performance of e-PP’s website. First, FDEMATEL was applied to construct interrelationships between evaluation criteria. Second, the weights of evaluation criteria were determined through FANP. The results show that ‘standardisation’, ‘links’, ‘reliability’ and ‘navigability’ are the most important criteria for the evaluation of e-PP’s website. A further analysis with WVA suggested managerial actions based on two-dimensional maps for improving website quality with limited resources. Different stakeholders, who are citizens, public authorities and suppliers, evaluated the website from their own perspective by using the proposed model. Although citizens only monitor the procurement process, public authorities and suppliers are the users of the e-PP’s website. In order to build an ideal website, designers must be aware of the users’ and citizens’ expectations, respectively. According to the users’ evaluation, it is urgent to


provide more sophisticated search tools and links to legal documents or forms concerning the procedure of procurement law. In addition, relevant and complete information could be reached easily in the website. The website in which current aspects satisfy users should assure the secrecy of authorities’ and suppliers’ data. Resources which are located for ‘attractiveness’ and ‘responsiveness’ must be redeployed for the efficient web-based public procurement.

This study provides important indications of the expectations of e-PP by all participants. Managers who have technical oversight for the web-enabled system will use the results from the website evaluation in different ways. As public authorities become more sophisticated and as users become accustomed to new types of e-government applications, their expectations will develop gradually. As a result, the applicability and relevance of the e-government applications and satisfaction level of users should be subjected to continuous reflection and evaluation by participants.

This paper shows that application of the FDEMATEL–FANP method to evaluate website is convenient and gives profitable opinions. Future researches may seek to apply this methodology to a variety of application areas to measure its effectiveness and usability. At the same time, researchers can use different methods to carry out the experiment, and compare the differences among them.

References Baker, R.J. (1999) ‘E-purchasing: big benefits, tough choices’, Global Purchasing and Supply

Chain Strategies, World Markets Research Centre, London. Brown, M.M. and Brudney, J.L. (2001) ‘Achieving advanced electronic government services:

an examination of obstacles and implications from an international perspective’, The National Public Management Research Conference, Bloomington, IN.

Büyüközkan, G. and Çiftçi, G. (2012) ‘A combined fuzzy AHP and fuzzy TOPSIS based strategic analysis of electronic service quality in healthcare industry’, Expert Systems with Applications, Vol. 39, pp.2341–2354.

Callender, G. and Schapper, P. (2003) ‘Public procurement reform in Australia: a Federal-State Evaluation, International Research Study of Public Procurement’, Proceedings of International Research Study of Public Procurement, Budapest, Hungary.

Chang, B., Chang, C.W. and Wu, C.H. (2011) ‘Fuzzy DEMATEL method for developing supplier selection criteria’, Expert Systems with Applications, Vol. 38, No. 3, pp.1850–1858.

Chen, T. and Sun, K.S. (2012) ‘Exploring the strategy to improve senior citizens’ participations on recreational sports’, Knowledge-Based Systems, Vol. 26, pp.86–92.

Chiu, Y.J., Chen, H.C., Tzeng, G.H. and Shyu, J.Z. (2006) ‘Marketing strategy based on customer behavior for the LCD-TV’, International Journal and Decision Making, Vol. 7, Nos. 2–3, pp.143–165.

Croom, S. and Brandon-Jones, A. (2004) ‘E-procurement: key issues in e-procurement implementation and operation in the public sector’, 13th International Purchasing & Supply Education & Research Association (IPSERA) Conference, Catania, Italy.

Dağdeviren, M. (2010) ‘A hybrid multi-criteria decision-making model for personnel selection in manufacturing systems’, J. Intell. Manuf., Vol. 21, pp.451–460.

EKAP (2012) https://ekap.kik.gov.tr/EKAP/Default.aspx?ReturnUrl=%2fekap%2f (Access date: 11/01/2012).

Electronic Signature Law (2010) Official Gazette No.: 25355, Published: 23/01/2004, Law No. 5070, Available from: http://en.hukuki.net/index.php?topic=1300.0 (Accessed 16 April 2012).


EUROPA (2004) Consolidated Directive (2004/18/EC) http://eur-lex.europa.eu/LexUriServ/ LexUriServ.do?uri=CONSLEG:2004L0018:20090101:EN:PDF (Access date: 10/31/2012).

Fassnacht, M. and Koese, I. (2006) ‘Quality of electronic service: conceptualizing and testing a hierarchical model’, Journal of Service Research, Vol. 9, No. 1, pp.19–37.

Fontela, E. and Gabus, A. (1974) DEMATEL, Innovative Methods, Report No. 2 Structural Analysis of the World Problematique, Battelle Geneva Research Institute, pp.67–69.

Hori, S. and Shimizu, Y. (1999) ‘Designing methods of human interface for supervisory control systems’, Control Engineering Practice, Vol. 7, No. 11, pp.1413–1419.

Jassbi, J., Mohamadnejad, F. and Nasrollahzadeh, H. (2011) ‘A fuzzy DEMATEL framework for modeling cause and effect relationships of strategy map’, Expert Systems with Applications, Vol. 38, pp.5967–5973.

Kabak, M., Burmaoğlu, S. and Kazançoğlu, Y. (2012) ‘A fuzzy hybrid MCDM approach for professional selection’, Expert Systems with Applications, Vol. 39, pp.3516–3525.

Kaisara, G. and Pather, S. (2011) ‘The e-government evaluation challenge: a South African Batho Pele-aligned service quality approach’, Government Information Quarterly, Vol. 28, pp.211–221.

Kaliannan, M., Awang, H. and Raman, M. (2009) ‘Government purchasing: a review of e-procurement system in Malaysia’, The Journal of Knowledge Economy& Knowledge Management, Vol. 4, pp.27–41.

Kaliannan, M., Chandran, S.D. and Hashim, R. (2010) ‘Electronic procurement implementation in Malaysia: suppliers’ readiness’, International Conference on Science and Social Research, Kuala Lumpur, Malaysia.

Kim, H.H., Kim, J.H. and Lennon, S.J. (2006) ‘Online service attributes available on apparel retail websites: an E-S-QUAL approach’, Managing Service Quality, Vol. 16, No. 1, pp.863–881.

Li, C.W. and Tzeng, G.H. (2009) ‘Identification of a threshold value for the DEMATEL method using the maximum mean de-entropy algorithm to find critical services provided by a semiconductor intellectual property mall’, Expert Systems with Applications, Vol. 36, pp.9891–9898.

Lin, Y.T., Yang, Y.H., Kang, J.S. and Yu, H.C. (2011) ‘Using DEMATEL method to explore the core competences and causal effect of the IC design service company: an empirical case study’, Expert Systems with Applications, Vol. 38, No. 5, pp.6262–6268.

Liou, J.J.H. (2012) ‘Developing an integrated model for the selection of strategic alliance partners in the airline industry’, Knowledge-Based Systems, Vol. 28, pp.59–67.

Liou, J.J.H., Tzeng, G.H. and Chang, H.C. (2007) ‘Airline safety measurement using a novel hybrid model’, Journal of Air Transport Management, Vol. 13, No. 4, pp.243–249.

Martilla, J.A. and James, J.C. (1977) ‘Importance-performance analysis’, Journal of Marketing, Vol.41, No. 1, pp.77–79.

Meade, L.M. and Sarkis, J. (1999) ‘Analyzing organizational project alternatives for agile manufacturing processes: An analytical network approach’, International Journal of Production Research, Vol. 37, pp.241–261.

Means, G. and Schneider, D. (2000) Meta-capitalism: The e-Business Revolution and the Design of 21st Century Companies and Markets, John Wiley & Sons Inc., New York.

Moe, C.E. (2010) ‘Procurement of IS in the public sector – a literature review’, 33rd Information Research in Scandinavia Seminar, Aalborg, Denmark.

Mokhtarian, M.N. (2011) ‘A note on developing global manager’s competencies using the fuzzy DEMATEL method’, Expert Systems with Applications, Vol. 38, No. 7, pp.9050–9051.

Onut, S., Kara, S.S. and Isık, E. (2009) ‘Long term supplier selection using a combined fuzzy MCDM approach: a case study for a telecommunication company’, Expert Systems with Applications, Vol. 36, pp.3887–3895.


Parasuraman, A., Zeithaml, V.A. and Malhotra, A. (2005) ‘E-S-QUAL: a multiple-item scale for assessing electronic service quality’, Journal of Service Research, Vol. 7, No. 3, pp.213–223.

Principles for Integrity in Public Procurement (2009) Available from: www.oecd.org/document/ 25/0,3343,en_2649_34135 _42768665_1_1_1_1,00.html (Accessed 3 April 2012).

Ramik, J. (2006) ‘A decision system using ANP and fuzzy inputs’, 12th International Conference on the Foundations and Applications of Utility, Risk and Decision Theory, Roma.

Rasheed, H.S. (2004) ‘Capital access barriers to public procurement performance: the moderating effects of ethnicity, gender and education’, Journal of Developmental Entrepreneurship, Vol. 9, No. 2, pp.22–43.

Republic of Turkey Public Procurement Authority (2009) Public Procurement System in Turkey, Kyrgyzstan, Available from: http://siteresources.worldbank.org/INTECACOSUPROC/ Resources/776023-1271709540888/6982537-1271775045268/Turkey.ppt (Accessed 16 April 2012).

Saaty, T.L. (1996) Decision Making with Dependence and Feedback: The Analytic Network Process, RWS Publications, Pittsburgh.

Saaty, T.L. and Vargas, L.G. (1998) ‘Diagnosis with dependent symptoms: Bayes theorem and the analytic hierarchy process’, Operations Research, Vol. 46, No. 4, pp.491–502.

Schapper, P.R., Malta, J.N.V. and Gilbert, D. (2006) ‘An analytical framework for the management and reform of public procurement’, Journal of Public Procurement, Vol. 6, Nos. 1–3, pp.1–26.

Schoenherr, T. and Tummala, V.M.R. (2007) ‘Electronic procurement: a structured literature review and directions for future research’, Int. J. Procurement Management, Vol. 1, Nos. 1–2, pp.8–37.

Seyed-Hosseini, S.M., Safaei, N. and Asgharpour, M.J. (2005) ‘Reprioritization of failures in a system failure mode and effects analysis by decision making trial and evaluation laboratory technique’, Reliability Engineering and System Safety, Vol. 91, No. 8, pp.872–881.

Shieh, J.I., Wub, H.H. and Huang, K.K. (2010) ‘A DEMATEL method in identifying key success factors of hospital service quality’, Knowledge-Based Systems, Vol. 23, pp.277–282.

Suri, P.K. and Sushil (2011) ‘Multi-perspective analysis of e-governance performance: a study of select agriculture related projects in India’, Journal International Journal of Electronic Governance, Vol. 4, No. 3, pp.259–272.

Thai, K.V. (2001) ‘Public procurement re-examined’, Journal of Public Procurement, Vol. 1 No. 1, pp.9–50.

Tooli, F.H., Omidian, A. and Fathi, M.R. (2011) ‘Applying fuzzy AHP and VIKOR to research supervisor selection: a case study’, American Journal of Scientific Research, Vol. 30, pp.19–27.

Tsai, W.H. and Chou, W.C. (2009) ‘Selecting management systems for sustainable development in SMEs: a novel hybrid model based on DEMATEL, ANP, and ZOGP’, Expert Systems with Applications, Vol. 36, pp.1444–1458.

Tsai, W.H., Chou, W.C. and Hsu, W. (2008) ‘The sustainability balanced scorecard as a framework for selecting socially responsible investment: an effective MCDM model’, Journal of the Operational Research Society, Vol. 60, pp.1396–1410.

Tsai, W.H., Chou, W.C. and Lai, C.W. (2010) ‘An effective model and development analysis for national park websites: a case study of Taiwan’, Tourism Management, Vol. 31, pp.936–952.

Tsai, W.H., Chou, W.C. and Leu, J.D. (2011) ‘An effectiveness evaluation model for the web-based marketing of the airline industry’, Expert Systems with Applications, Vol. 38, pp.15499–15516.

Tzeng, G.H., Chiang, C.H. and Li, C.W. (2007) ‘Evaluating intertwined effects in e-learning programs: a novel hybrid MCDM model based on factor analysis and DEMATEL’, Expert Systems with Applications, Vol. 32, No. 4, pp.1028–1044.


Wittig, W.A. (2003) ‘Public procurement and the development agenda’, United Nations Conference on Trade and Development, International Trade Center, Geneva, Switzerland.

Wu, W.W. and Lee, Y.T. (2007) ‘Developing global managers’ competencies using the fuzzy DEMATEL method’, Expert Systems with Applications, Vol. 32, No. 2, pp.499–507.

Yadav, N. and Yadav, H. (2009) ‘An electronic government model based on case study approach’, Electronic Government, An International Journal, Vol. 6, No. 4, pp.421–432.

Documents

A holistic evaluation of the e-procurement website by using a hybrid MCDM methodology