Proceedings of the 16th International IEEE Annual Conference on

Intelligent Transportation Systems (ITSC 2013), The Hague, The

Netherlands, October 6-9, 2013

MoC9.4

Assessment of Maturity and Efficacy of Toll Collection Ecosystems

Adriano Galindo Leal, Member, IEEE, Alessandro Santiago dos Santos, Mario Yoshikazu Miyake, Member, IEEE, and Claudio Luiz Marte

Ahstract- Control Objectives for Information and related Technology (CobIT) establish maturity models, the assessment of process capability is an essential part of IT governance implementation. In an analogous manner, the efficacy of operations in Toll Collection Ecosystem could be evaluated using the same approach from CobIT. Maturity models enable managers to identify gaps in key processes and controls. It describes a tool to assess the maturity and effectiveness of all processes associated with the Toll Collection Ecosystem. The Toll Collection processes constitute an ecosystem that involves the quality and maturity of operations, business processes, institutional aspects, equipment maintenance and infrastructure management. The creation of a methodology for effectiveness and maturity analysis of the full Toll Collection Ecosystem allows the establishment of quantitative and qualitative parameters, which may be assessed and monitored, therefore, enabling a useful tool for the operators' and Government Regulatory Agencies' decision-making processes.

I. INTRODUCTION

To increase the speed of investments in the modernization of road infrastructure and improve the quality of services to users, following a worldwide trend, some Brazilian States initiated the transfer of operations of key highways to private companies. To support this transition, it was necessary to set up an organization to serve not only as a watchdog, but also as a mediator between the public authorities (State Government), the Highway Operators, and users of the highways network. This was the case in the state of Sao Paulo, the first in Brazil to establish a comprehensive program of concessions of major highways operations

In the last decades, the investments of the Government of the State of Sao Paulo in the state highway system were not enough to keep pace with growth on traffic volumes and the changing patterns of road transport business models.

Adriano Galindo Leal, Member, IEEE, Alessandro Santiago dos Santos, Jose Augusto de Oliveira, Jose Rita Moreira, Edson Pistoni and Mario Yoshikazu Miyake, Member, IEEE, are with the Center for Information Technology, Automation and Mobility. IPT - Institute for Technological Research at Av. Prof. Almeida Prado, 532 - Cid. Universitaria - Sao Paulo/SP - Brazil (+55 (11) 3767-4748; fax: :+55 (11) 3767-4285; e-mail: leal@ipt.br or leal@ieee.org).

Claudio Luiz Marte, is with University of Sao Paulo, Cidade Universitaria - Sao Paulo/SP - Brazil (e-mail: claudio.marte@poli.usp.br).

978-1-4799-2914-613/$31.00 ©2013 IEEE 523

Therefore, in 1998, a concession program was shaped to transfer the operation to private companies. Furthermore, in 2002, the Regulatory Agency for Delegated Transportation Services of the State of Sao Paulo-ARTESP was created to ensure compliance of the concessionaires with the rules and regulations governing Public Utility Services operations and ensure the fulfillment of contracts signed between the State and the pri vate operator.

Efficacy in operation and management of highways is among key requirements of concession contracts, which include a variety of commitments agreed in public-private partnerships-PPPs.

Thus, regulatory agencies (such ARTESP) face the challenge of regularly measure the actual levels of services delivered and confront them with what was established in the concession contract [1]. In [2], we analyze the pitfalls and difficulties of running and maintaining quality assessment programs of urban transportation systems, which may take years before its full implementation.

The ARTESP commissioned IPT to build a methodology for assessment of the effectiveness and maturity level of all processes related to the Toll Collections System of all concessionaires. The methodology evaluates the performance of each operator by two different points of view: from users of conceded roads and the regulatory agency.

This paper presents the methodology developed for the evaluation and governance model of Toll Collection System, based on best practices proposed by the Control Objectives for Information and related Technology (CobIT) [3], guide recommended by the Information Systems Audit and Control Association (lSACA) for the Management of Information Technology (IT). This approach allows a fast implementation so that the evaluation system costs do not compromise benefits.

The methodology takes into account the business objectives of the concessionary, rules and regulations concerning highway operations, also defines the assessment criteria, performance indicators, and helps to identify capability gaps in evaluated systems.

This paper is organized in six sections. Section IT describes the maturity models from CobIT. Section III describes the reference model for the assessment of effectiveness of services provided by operators of toll collection systems in highways.

Section IV describes how the evaluation process uses a bottom-up approach starting with activities that are gathered in processes that constitute domains one level up to compose the overall operation at the top. Section V presents the results from the application of the proposed model on twelve Highway Operators from the Sao Paulo state highway system, from 2008 to 2012. Finally, Section VI presents the conclusions and future works.

II. MATURITY MODELS

According to CobIT maturity models [3], the assessment of process capability is an essential part of IT governance implementation. In an analogous manner, the efficacy of operation for Toll Collection Ecosystem could be evaluated using the technique from CobIT.

"Maturity modeling for management and control over IT processes is based on a method of evaluating the organization, so it can be rated from a maturity level of non-existent (0) to optimized (5). This approach is derived from the maturity model that the Software Engineering Institute (SEl) defined for the maturity of software development experience. Although concepts of the SEI system were followed, the CobIT implementation differs significantly from the original SEI, which was oriented toward software product engineering principles, organizations striving for excellence in these areas and formal evaluation of maturity levels so that software developers could be 'certified'. In CobIT, a broad definition is provided for the CobIT maturity scale, which is similar to CMM, but interpreted for the nature of CobIT's IT management processes. " [3]

The methodology follows the same basic principles and uses the same scale to measure the maturity level (Figure 1).

Initiall Repeatable Defined Managed and Non-existent Ad Hoc but Intu itive Process Measurable Optimised

o 2

I

LEGEND FOR SYMBOLS USED

Enterprise current status

..... Industry average

Enterprise target

3 4 5

I I I

LEGEND FOR RANKINGS USED

�Managemenl processes are nol applied al all. I-Processes are ad hoc and disorganised.

2-Processes follow a regular pattern. 3-Processes are documented and communicated.

4-Processes are monilored and measured.

5-Good praclices are followed and automaled.

Figure 1. Reference Model. Source: [3]

However, the objectives controls, domains, and activities for the management of the toll collection ecosystem is not complete adherent with IT environment; thus, an adjustment was essential. In this sense, the next section presents an overview of the main changes, which are specific on [4].

978-1-4799-2914-613/$31.00 ©2013 IEEE 524

III. REFERENCE MODEL

The methodology follows the same principles and guidelines sponsored by CobIT, assimilating the method of aggregating Activities in Processes, Processes in Domains, and Domains in the overall approach to be evaluated. Balanced scorecard techniques are used to quantify the scores for all assessments made in the evaluation process. the proposed method was applied on 12's operators of Sao Paulo State, allowed the creation benchmarking their performance against each other and with a reference level of quality defined by ARTESP.

� B u

c � it: � 0 .p 1Jl (lJ '" (lJ u:: '" E .::: (lJ .2 E � .r::: �

E

Figure 2. Reference Model. Source: Adapted from [3]

As seen on [3] and [4], all the activities are clustered in business processes using the same technique used on CobIT. These faces are detailed on subsections A, B and C.

A. Model Structure - Resources

The resources defined in this model include vehicle fleet, Information, infrastructure from toll systems (especially IT infrastructure) and People.

Infrastructure resources include not only equipment (hardware elements), but also elements such as data, manuals, buildings, furniture, security, maintenance, cleaning, construction works, trammg and written procedures, as well as the technical team responsible for maintaining operational this infrastructure .

B. Model Structure - Criteria

On the upper side of the cube are the Criteria by which acti vities are evaluated, borrowed from CobIT and adapted for the transportation sector environment: Effectiveness, Efficiency, Confidentiality, Integrity/ Authenticity, Availability, Safety, Comfort/user satisfaction, and Compliance.

C. Model Structure - Domains, Processes and Activities

This structure contains five Domains of analysis. A domain is a group of process control variables, each incorporating a group of discrete activities. Thus, we selected five (05) Domains to covers all the processes that give support to the Toll Collection System: Institutional,

Operational, Toll Collection, Maintenance and Infrastructure (Table 1).

TABLE I - DOMAINS. AMOUNT OF PROCESS AND ACTIVITIES

Domains Quantity of Processes Quantity of Activities

Institutional 5 20

Operational 6 19

Maintenance 4 13

Toll Collection 4 16

Infrastructure 4 18

Total 23 86

In this paper, a Process is an aggregation of crucial activities for management and operation of Toll Collection Systems. The complete evaluation system is composed of 86 activities. The definition of how these activities must be evaluated and graded represents the major contribution of this project. They are based on the regulations and rules described on contracts and agreements between operators and ARTESP. Additionally, they embody the knowledge accrued from previous collaborative works done by IPT and ARTESP in the period from 2002 to 2008.

For demonstration purpose, the next sections of this paper will discuss the leading Domains, some of its constituent processes and their associated activities.

1) Institutional Domain

In this domain, all formal procedures in the company concerning the Toll Collection System are analyzed. The focus is on the organizational structure adopted for the management of operation, maintenance and toll collection at the toll plazas.

The model of governance and the organizational culture, which support and interact with the Toll Collection Systems, are critical to guarantee effectiveness and quality of services provided. It also addresses user needs for comfort and safety, and commitments made with the conceding authority.

2) Infrastructure Domain

In this domain, the assumption is that the toll plaza infrastructure is considered a public property and it must be kept in proper operating condition and, at the end of the concession period, must be returned in perfect condition. The perceived quality of service is closely tied to the condition of the toll facilities, signaling system and plaza.

3) Operation Domain

In this domain are evaluated the actions, procedures and operational strategies of the operator to support Toll Collection operation and the smooth flow of traffic, guaranteeing user comfort and safety at the toll plazas.

The proper operation of the plazas is one of the cornerstones in management of traffic flow on the highway, where the operational model aims at meeting demand, at different traffic volumes, and being ready to assist

contingency processes, always taking into account the best practices and superior customer care.

4) Maintenance Domain

In this domain, it is performed the analysis of management and maintenance of the service structure at the toll plazas. As premises, all equipment must be kept in proper operational status, taking into account the operations standards and the technological revitalization policies of systems and equipment installed in toll plazas.

The assessment of the management of maintenance services highlights the obsolescence of electronic resources, as well as any deviations from the obligation to maintain and update the public infrastructure.

5) Toll Collection Domain

This Domain focus on the Toll Collection business processes adopted by the Highway Operator, which includes policies, procedures, systems and operations at the toll plazas. It evaluates the consistency of activities - from Level 1 (Traffic Lanes), through Level 2 (Operational Control Facility at the Toll Plaza) to arrive at Level 3 (Toll Collection Control Center), which has a consolidated view of all toll plazas of the Highway Operator.

Attention is paid to the operator' s capability to generate revenues to cover operational and maintenance costs, as well as ensure the financial health of the Highway Operator.

IV. EFFICACY EVALUATION MODEL

The proposed model is implemented with a bottom-up approach. The evaluation process starts with activities, then goes up to processes, domains and the whole entity.

� n [ Domains

\0,-------------,

Score of Domains based in formula using level of importance of the process to domains.

Figure 3.

Score of processes All specialists based in formula evaluate the using level of the activities based importance of in specific activity to process. criteria

Evaluation process diagram

Each activity is evaluated according to an appropriate sub set of the criteria defined for the model. The evaluation process is based in four "quality standards". These levels reflect the quality of execution of the activity by the service operator: "c" (critical/score 0); "i" (insufficienUscore 3); "s" (suitable/score 7) and "b" (better than expected/score 10).

At the time of the evaluation of a given activity, the analyst will pay attention only to the criteria under which it should be evaluated. Any deviation from the middle score, in the sense of excess (b) or deficiency of quality (c or i) is

978-1-4799-2914-613/$31.00 ©2013 IEEE 525

annotated and commented upon, in order to characterize the score deviation from the expected value. The analyst may add photos to illustrate his comments.

Once all the activities are evaluated, the process goes one­step further, to the level of processes.

Each process is attached to a sub set of activities. The score of the process is calculated adding the scores of each related activity multiplied by an importance weight. The sum of importance weights for the process should equal one.

Score of Process (k) =

2:1=1 Activity importance weight(i)* Activity score(i) 2:(=1 Activity importance weight(i)

j represents the number of activities in process (k)

(1)

In a similar way, the score of one domain is calculated from the scores of processes attached to it:

Score of Domain (k) =

2:1-1 Process importance weight (i) * Process score(i) 2:(=1 Process importance weigfit(i) (2)

j represents the number of processes in Domain(k)

Finally, the score of transport Highway Operator IS derived from the scores of all domains:

2:1-1 Domain importance weight (i) * Domain score(i) 2:(=1 Domain Importance welgfit(i)

j = 5(domains).

(3)

To illustrate the results from the evaluation process, Figure 4 presents an example for score propagation from the bottom-up approach to Operation Domain.

Figure 4. Evaluation process diagram

The methodology presented in this paper allows the assessment of Highway Operators based on a single model. The result of the evaluation provides an operational overview of the quality offered by the Highway Operators, which will serve as a basis for decision -making by Government Agencies, as well as a valuable input for quality improvement programs in areas with critical or insufficient scores.

QJ .... o u VI !: o

'';::: ItI

-= ItI �

Institutional; 2,20

Operational; Finantial; 4,30 4,20

Maintenance; 1,60

Infrastructure; 1,63

Figure 5. Representation of the Domains Evaluations'

·3,63

.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! •••• II!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!I ·6.52

• 2,60

••••••••••••••••••• 5,29

• I-Institutional _ 2-operation .3-Maintenance _ 4-Toll collection _ 5-lnfrastructure

Figure 6. Results by Domains

This methodology help decision makers to understand the quality of services under different points of view with the granularity needed (Figure 3 and 4).

As an example, analyting the domains' scores in figure 5, it should be noticed the poor performance in two domains: maintenance and infrastructure. According to the findings in this evaluation, the processes of maintenance should gain greater strategic importance, being the area more lacking in good practices, and with lower priority compared to other domains.

The operator would benefit from the adoption of solid maintenance methodologies, such as the control of equipment life cycle and Reliability-Centered Maintenance, both within a solid strategic program for sustainable development, in order to achieve measurable results in increased productivity, energy efficiency, waste utilization, decrease in pollution levels, process improvement, and even in launching new products [5].

Each employee requires, in addition to "know-how", "the desire to do". Consequently, someone will want to do something only when he or she consciously understands the importance of his contribution to the whole. In resume, the more he learns and understands, more responsible he feels [6].

1 Obs: The LARGER the number of critical or insufficient evaluations, the GREATER will be the size of the sphere. The BETTER the score, the HIGHEST will be the position of the sphere.

978-1-4799-2914-613/$31.00 ©2013 IEEE 526

V. DEPLOYMENT AND EXPERIMENTATION IN REAL CASE

Under the program of road concessions in the State of Sao Paulo, Brazil, almost 5,400 kilometers of highways began to be managed, operated and economically exploited by the private initiative. These concessions are responsible for a large percentage of the Average Daily Volume (ADY) and, consequently, are of fundamental importance in the movement of goods and people in the State of Sao Paulo.

The methodology presented in this article consolidates and describes in detail the aspects discussed in [4] and [7], and has been applied in 12 Highway Operators in the State of Sao Paulo during the period from February 2008 to January 2012.

This methodology was applied to toll collection systems of each of those 12 Highway Operators in two weeks visit to the premises of toll plazas.

The evaluation process included face-to-face interviews with managers and operators of the toll collection system, assessment of operational actions and events, and assessment of other systems that interact with it.

The final scores of assessment received by the twelve operators, referring to the five Domains analyzed (institutional, operation, maintenance, toll collection and infrastructure), represents the overall score obtained by the operator in the quality analysis of the toll collection system.

During the evaluation, it was realized the need to return to the site of inspection after a period of at least one year. On

this occasion, besides allowing the Highway Operators show the measures adopted to solve the problems pointed out in the First Phase of the Project, it was evaluated the effectiveness of all activities through new field interviews, personal observations, and analysis of the "declaration of Evolution". In this document, the Highway Operator lists the steps taken to solve the deficiencies found in the previous phase. Each activity is scored according to the effectiveness of the action taken in one of the following conditions: Problem Solved, Problem Partly Solved, No action, and Degraded.

On Phase 1, 1032 activities were analyzed during the first visit to all Highway Operators, and 381 were considered problematic (Insufficient or Critical).

Problematic Actions plan activities /?f

26%

� Results of actions In

.,Phase 2

'V 37%

1% • Better tlian expected DSuitable o Insuficient . Critical

• Partialy solved • Nothing • Degrated

Figure 7. Comparison between Phases 1 and 2

Observing the numbers in Figure 7, we can see that on the second visit, Phase 2, there was improvements in 73% of those 381 activities, and 37% (142) were fully resolved, demonstrating the methodology usefulness. Therefore, we can conclude that this project has contributed significantly to the identification and improvement of quality and effectiveness of activities.

VI. CONCLUSION

This paper presents a methodology for the evaluation of

the effectiveness of toll collection systems, as implemented

by highways networks operators under concession contracts

with the state of Sao Paulo Government.

The application of this methodology on Highway

Operators provides a quality benchmark of the services

offered to the highway user population.

Given the goals for quality levels established by ARTESP,

the assessment results give a clear picture of the attention

areas for each operator in the continuous improvement

program they should adopt to reach their targeted goals.

This methodology also enables the Highway Operator to

evaluate their business. Consequently, they can identify,

improve, and/or modify any weaknesses in the various areas,

not only to be in accordance with regulation constraints.

With regular checks are expected not only improvements

in the effectiveness of the activities, but also in the

management of business processes. With this, it is expected a

greater maturity, professionalism of staff and a reduction in

operating costs over the long term. This is beneficial not only

to the government or to the utility, but also to the whole

society. Another byproduct is the possibility of the

improvement of regulation requirements applied to Highway

conceSSIOns.

The model holistically observes all the previously

mentioned Domains involved in the Toll Collection System.

The CobIT maturity model was adapted to meet the

evaluation needs, to cope with the accumulated knowledge

base from previous assessments, for process definitions and

the specification of the dimensions of analysis of capacity

maturity by domain in future assessments.

For each domain, the following dimensions are analyzed:

Awareness, Policies, Technology, skills, responsibility, and

goals of each Business Process [8]. At this time, April 2013,

this model was fully applied for 12 utilities and, until 2014,

will be applied for seven more.

In future works, it will be analyzed the influence of

process maturity level on effort, cost and quality. Therefore,

we can have a clear picture of how much influence the

maturity level has on staff turnover, costs, infrastructure

degradation, and compliance with contracts and Regulatory

Agency Rules [9].

978-1-4799-2914-613/$31.00 ©2013 IEEE 527

ACKNOWLEDGMENT

The authors would like to express their deepest gratitude and appreciation to: Fernando Jose Gomes Landgraf, Carlos Daher Padovezi, Zehbour Panossian, Maria Rosilene Ferreira, Alex Fedozzi ValIone, Cleudicir Jose Gaspar, Gilberto Abrahao Martins, Jose Augusto de Oliveira, Jose Rita Moreira, Jose Wellington de Oliveira Miranda, Omar Joaquim de Castro Martins, Alais Antonio TelIes, Maria Spidalieri, Livia Cabral de Souza Rodrigues, PamelIa de Moura Cruz, Pedro e Dirce Apparecida Chinelato, Maria Aparecida Leal and Yasmin Quintino Santiago.

Additionally, the authors would like to extend our deepest gratitude and appreciation to ARTESP and his team: Karla Bertocco Trindade, Giovanni Pengue Filho, Jose Carlos de Faria Vieira, Rui Fernando Nobrega Gouveia, Luiz Antonio Carazzi e Anderson Hale Rodrigues.

REFERENCES

[1] V. Liebert e H.-M. Niemeier, "Benchmarking of Airports - A Critical Assessment," em Proceeding 17th WCTR, Lisbon, 2010.

[2] M. Boltze e H. Jentsch, "Integrated quality management for urban transport systems," em Proceedings 17th WCTR, Lisbon, 2010.

[3] IT Governance Institute, CobIT 4.1, vol. I, ISACA, 2007.

[4] A. Santiago, A. Saleta, R. Gouveia e A. Rigo, "Metodologia de avalia�ao de sistemas ITS baseada em normas de governan�a com enfase na qualidade do sistema de arrecadayao do pedagio," Congresso Brasileiro de Rodovias e Concessoes, Exposir;:iio lnternacional de Produtos para Rodovias, p. 10, Junho 2009.

[5] Secretaria Tecnica do Fundo Verde-Amarelo, "Programa de desenvolvimento de Recursos Humanos para atividades estrategicas em apoio 11 inova9ao tecnol6gica (RHAE-Inovayao ).," Brasil, 2011.

[6] F. M. &. M. S. I. Fritzen, "Treinamento como Pratica de Sustentabilidade: Urn Estudo de Caso sobre 0 Processo de Implanta�ao da ISO 14001 na Refinaria Ipiranga SA," 2008. [Online]. Available: http://www.unifae.br/publicacoeslpdf/sustentabilidade/fabiano_suzana.pdf. [Acesso em 11 04 2013].

[7] A. G. Leal, A. S. Santos, J. A. Oliveira, O. Martins, G. Martins, C. Gaspar, R. Gouveia e J. R. Moreira, "Aplicayao de Melhores Praticas na Gestiio do Ecossistema de Arrecada9ao de Pedagio nas Concessionarias de Rodovias," em 70 Congresso Brasileiro de Rodovias e Concessoes CBRC, Foz de Igua9u, 2011.

[8] G. L. G. Roger Debreceny, "IT Governance Drivers of Process Maturity," 2011. [Online]. Available: http://jebcl.comlsymposium/wp­content/uploadsl2011108/Governance-Process-Maturity.pdf. [Acesso em 2012 10 27].

[9] P. Marija Andjelkovic Pesic, "Business Process Management Maturity Model and Six Sigma: An Integrated Approach for Easier Networking," 2009. [Online]. Available: http://emnet.univie.ac.at/u pi oadsl medial And j elkovi c -Pesic _01. pdf. [Acesso em 27 10 2012].

978-1-4799-2914-613/$31.00 ©2013 IEEE 528