TM 204 PREGINET Complete Paper + Bianca + Bong + Cha

8/7/2019 TM 204 PREGINET Complete Paper + Bianca + Bong + Cha

1/32


2/32

A N E VA L U AT I O N R E P O RT O N T H E

Philippine Research, Education andGovernment Institution NETwork

(PREGINET)

Submitted By:

Alvarez, Ma ry. JoanCuevas, Juan Ce a sar M.

Derecho, Charisse C.

October 2010


3/32

Table of Contents

Topics Chapter

PageNo.

I. Objectives of the Evaluation 1 1 to 2II. Background of the PREGINET Project 2 3

III. Evaluation of the PREGINET Project 3 4A. Problem Identification and ProposalStage 4 5

B. Implementation Stage 5 6 to 8C. Post- Evaluation Stage 6 9

IV. Conclusion 7 10 to 12V. List of ReferencesAppendices 8

Objectives of the Evaluation

his evaluation report aims to assess the Philippine Research, Education andGovernment Institution NETwork (PREGINET) project as a research anddevelopment (R&D) project focusing only on the aspects that has direct

impact to research and developmentR&D .TSpecifically, the objectives are to :

To Specific, Measurable, Attainable, Relevant and Time-bound

Dd etermine the concrete research and development (R&D) R&D output /s of the project ;

To Dd etermine how effective the Assess the PREGINET project , PREGINET, interms of project phases: Planning, Implementation and Post Implementation

1


4/32

was managed from the planning and the proposal stage , to theimplementation and post-evaluation stages and how each of these stages contributed to the success or failure of the project , and in terms of R&D

To Assess the impact of PREGINET in terms of its R&D objectives and output

Assess the impact of PREGINET in terms of its contribution to theadvan cement of network technologies Pp rovide recommendations effectively

managefor the management of future DOST-funded R&D projects .

Background of the PREGINET Project

I

n 1996-1997, the strategy of the RPWEB RPWeb initiative was for governmentagencies to connect through commercial ISPs. Unfortunately, they experiencedvarying levels of service and there was a a lack of sufficient local connectivitybetween commercial ISPs.

During those times, There became a trend that industr ially-advanceiallyadvance d countries have embrac inged a knowledge economy and realizingrealized the value of setting-up Research and Education Networks (RENs) tofoster innovation. ASTI found th is e relevant to the Philippines and thus identifiedthe need for the Philippines country to have a network that is contiguous andwide in coverage, with available bandwidth on which next-generation networktechnology and services can be test inged and deploymentdeployed , and whereapplications development, experimentation, and demonstration can beconducted. The PREGINET project was conceptualized with all these in mind. T hehe project ,based on its documented research proposal, seekssought to accomplish thefollowing objectives :

ii

2


5/32

foster partnerships among government, academe, industry, and theresearch community all over the country ;

facilitate the introduction of next -generation network technology andservices by providing better network infrastructure and new services (e.g.security, quality of service, etc.) for research, education, and keygovernment institutions , and

said network will be used in the development and demonstration of novelapplications for science, education, health, and business

iii


6/32

P R E G I N E T E V A L U A T I O N R E P O R T

The PREGINET project details are summarized in TableFigure 1 below:

ProgramComprehensive Program to Enhance TechnologyEnterprises(DOST-COMPETE)

Project

Virtual Centre for Information Technology Innovation inInformation Technology (VCTI-IT): Development of a HighPerformance Research and Education Network: The PhilippineResearch, Education, and Government Information Network(PREGINET)

Duration

June 1, 2000 May 31 20051st Extension: June 1, 2005 May 31, 20062nd Extension: June 1, 2006 February 28, 20073rd Extension: March 1, 2007 December 31, 2007

Total ProjectCost Php78,887,793.10

ImplementingAgency

Advanced Science and Technology Institute Department of Science and Technology (ASTI-DOST)

MonitoringAgency

Philippine Council for Advanced Science and TechnologyResearch and Development (PCASTRD)

CooperatingAgencies DOST Agencies and Regional Offices, PREGINET Partners

Figure TableFigure 1 PREGINET Project Information

Evaluation of the PREGINET Project

TI

3


7/32


n order to determine how effective the Phe projectREGINET project was managed,it will be evaluated in terms of thein the context of each of the three projectmanagement stages in project management :

A. Problem Identification Concept DevelopmentConceptDevelopment aA nd Proposal Stage ;- Problem Identification and Solution Proposal- Project Objectives and Expected Output

B. Implementation Ss tage , and

- Research Outputs- Use of Funding- Staffing- Risk Management- Product Development

C. Post- iEmplementation evaluation E valuation stagStagee- Evaluation Team

- Evaluation Objectives - Evaluation Methods - DAP Evaluation Results Findings/ Results of DAP Evaluation - DAP Recommendations .

Problem IdentificationConcept Development andProposal Stage

I

n this section, we view the project in terms of its strategic justification 1 , whichrequires that the objectives of the project are : (1) are consistent with theagencys strategic objectives; 2) are are consistent with general Governmentpolicy; 3) can be justified on the grounds the market is failing to efficiently deliver

1 Department of Treasury and Finance Government of Western Australia, Project Evaluation Guidelines,(January 2002)

A


8/32


the proposed project outputs; and 4) take advantage of and has consideration forcurrent and expected future economic conditions.

The project implementerproponent , ASTI, essentially identified the followingproblems and/or needs that arewere to be solved addressed by the PREGINETproject:

Interconnectivity of government networks ; Advancement of ICT Information and Communication Technologies (ICT) in

the Philippines , and Network dedicated to research .

The problem identification process and the actual identification of the problemsidentified that are to be addressed by ASTI is are the basic initial weakness es ofinthe project. It is a case where the implementer proponent attempts to do toomuchalready has pre-conceived solution, which led to the flawed problemidentification process and in with inaa dequate screening and evaluation . A of a lternative solutions to the problem were not identified and there was no anassessment of whether the proposed solution is the best option to address theidentified problems was not conducted..

The first problem, interconnectivity of government networks, is out of scope of the institutes ASTIs mandate , which is to Conduct scientific research and development in the advanced fields of Information and CommunicationsTechnology and Microelectronics. Interconnectivity on a nationwide scope is theDOTCsDepartment of Transportation and Communications roleand is essentiallyDOTCs burden to bear . Given the projects initial timeframe of five ( 5 ) years, thecapabilities of commercial internet service providers ( ISPs) are bound to catch upand will be sufficient to meet the needs of non-research government firms. Thiscan be done without DOST-ASTI spending their R&D budget to subsidize thesegovernment agencies internet connection.

The second problem , unlike the first one, is in line with ASTIs mandate; however,the approach to the solution is again weak and should have been re-evaluated.The implementers proponents rationale goes: in order for us to research and

develop advanced networking technologies we need to first spend our budget ona network . If they had a clear R&D strategy with a project portfolio and strategyfor product development of advanced, undiscovered and cutting edgetechnologies then that will warrant the need for a high-bandwidth network.Unfortunately, the R&D strategy is also weak as will be described in latersections.


9/32


Suggested: The second and third problems are in line with ASTIs mandate;however, the approach to the identification of the solution is again weak andshould have been re-evaluated. The proponents rationale is :: in order for us topromote the advancement of ICT in the Philippines , we need to first to set-up adedicated network infrastructure for local research institutions to have access tointernational research papers ,; facilitate knowledge exchanges and facilitate enable the development, testing and demonstration of advanced networking technologies .

ASTI could have focused on determining ways to advance ICT in the Philippinesby creating a well-balanced project portfolio with clear R&D strategies for thedevelopment of advanced, undiscovered and cutting-edge technologies. Having ahigh-bandwidth network is neither a necessary pre-requisite nor the only meansto achieving the desired objectives and virtual research communities can beorganized even without setting-up the said network infrastructure. Unfortunately,the R&D strategy is also weak as will be described in later sections.

What could have been morewas more feasible and efficient to advance ICT in thePhilippines is towas to focus their efforts on purely R&Ddeveloping new networktechnologies projects and taking steps to ensure their commercialization .; Aanexample can be made of the Pawikan Project. 2 Pawikan is a network monitoringsoftware that was developed under PREGINET and was ready to be sold to a alocal telecomstelecommunications company. However, Dd ue to lack of properdocumentation and turnover of the source code, the knowledge left when thedeveloper resigned. What was left was a source code that no one in theorganization i wa s willing to study and the thus, the project wasabandonedfizzled .

To address the need for a research network MoreoverFurthermore , a researchnetwork run solely by a a private telecommunications company company wouldhavv e been cheaper as market competition between internet serviceprovidersavailable ISPs lowered the ratesdrives the rates down. Needless to say,and it would also serve s the same purpose as one that is administered byASTIPREGINET. In fact, PREGINET services are not as competitivecannot competein terms of ratesbecause they acquire the internet connection also through theseISPs and even the network reliability of the networkis reportedly problematic . Asfor researches , with the materials that ASTI can access via their partnership withAsian Internet Interconnection Initiatives (AI3)AI3 and Asia-Pacific AdvancedNetwork (APAN)APAN , they could have provide d in a web portal where localresearch agencies institutions can visit to gain access and thereby encouragemore participation and sharing of research outputs .

2Pawikan Network Monitor, http://pawikan.sourceforge.net/
http://pawikan.sourceforge.net/http://pawikan.sourceforge.net/


10/32


Objectives and Expected Output

In this section, the objectives and output are assessed. We look at how clearlythey were defined and how success and failure of these objectives will be usefulfor the proper evaluation of the project.

The project proposal states five main objectives and one generic statement forthe expected output (Figure 2).

Figure 2 PREGINET Project Objectives and Expected Output

The project proposal states five main objectives and 1 statement of expectedoutput (see figure 2).

On the surfaceIn the literal sense, the objectives may sound are clear andplausible. However, they are simply answerable with Yes/No and are notmeasurable or tangible by any means. Thus it is possible that practically anythingcan happen in the project and the objectives will still be met.

On the surface, the objectives may sound clear and plausible. However, they aresimply answerable with Yes/No and are neither measurable nor tangible. Thus, itis possible that anything can happen in the project and the objectives will still bemet.

To say that Tt he expected outputs are vague would be an understatement . Theexpected outputs and cannot even be seen or measured. Questions like How,


11/32


What, How many or How much should be answered by the expected outputs. Thefollowing are a few examples:

Attain X atotaln X amount of savings due to usage of PREGINET servicessuch as video conferencing and voice over internet protocol (VOIPVoIP) ingovernment agencies, instead of normal long-distance calls and travel ;

Reach an a X number of partners who ve published research papers inrefer re e d journals ;

Commercialize at least one ( 1 ) new networking technology per year , and Achieve a n X percentage measurable increase in researches published in

scientific papers , and , patents granted , etc., implementationsed and use of tilized network technologies in 5 years of operation

The objectives could have been patterned after SMART objectives, a mnemonicused in project management during the project objective - setting stage.SMARTSMART stands for S pecific, Measurable, Attainable, Relevant and Time-bound.

In view of the foregoing, Ww e can say that It is hard to say if this is intentional,but at this point in the planning process , the project should have been re jected orhave been returned for revision and further re -evaluat ioned or even rejected .

Implementation Stage

he project implementation framework is summed up by the diagram below(Figure 43 )(Appendix 1) . Given that DOST mandated the thrust towardsScience and Technology (S&T) development/advancement (?) and wants

sought to address the ICT needs of the Philippines, the projectthe intention is toaimed to utilize ICT technologies, Network network services and R&D on next-generation products to producein a high a high - performance Research andEducation networkREN, which is the ( Philippine Research, Education, andGovernment Information NETwork (PREGINET)PREGINET . The overall goal of

T

B


12/32


PREGINET the project is to establish a nationwide broadband network tointerconnect academic and research institutions as well as government agenciesin order to strengthen the country's science and technologyS&T capability andfurther to enable e-governance.

Figure 3 PREGINET Project Conceptual Framework

Its The projects direct beneficiaries are the following :

1. Information Technology IT Industry2. User Communities for Agriculture3. User Communities for Distance Education4. User Communities for Networking Technologies5. User Communities for Disaster Mitigation6. User Communities for Bioinformatics and Telemedicine

Based on the approved project proposal, Aa t the end of th ise project, the se werethe expected deliverables are :

1. Operational exchange point for research, education, and governmentnetwork interconnection ;. 2. Network testbed for research in and testing of next generationnetwork technologies, services, and equipment, and for the development,


13/32


experimentation, and demonstration of novel applications in the areas of e-commerce, telemedicine, distance education, among others ;.3. Competency in network equipment testing and integration services ;4. . Competency in network technology integration and networkservices provisioning ;.5. Developed and demonstrated novel applications for e-commerce,telemedicine, distance education, and other areas , and.6. Connectivity to research networks outside the country.

Analysis on Implementation Stage

The analysis of the projects iI mplementation stage focuses on major areas thatare crucial to the success or failure of PREGINET as a n R&D Research and

Education network Development organizationproject . The areas for analysis arethe following :

Research Outputs Return of Investment Staffing Risk Management Product Development


14/32


Research Outputs

Included in the intended deliverables of this project are the establishment of a: Nn etwork testbed for research in and testing of next -generation networktechnologies, services, and equipment , and for the development,experimentation, and demonstration of novel applications in the areas of e-commerce, telemedicine, distance education, among others.

However , at the assessment of the final status of the project, it was discoveredthat no new generation technologies w ereas developed, but instead, it wasmerely adaptation adoption of matured technologies made developedabroadreadily available in the market .

In addition, Aa mong the functions for the network testbed would be to alloware toenable experimentation and testing of new technologies and equipment (e.g.ATM, wireless broadband technologies, IP over SONET/SDH, DSL technologies,etc). However, since these technologies can also be tested usingcommericialcommercial ISPs , it would have been a more cost-efficient strategy if the project did not spend for the setting-up ofoffset-up the network infrastructureand instead also, the project should have been directed at testing thesetechnolouse d the services of cheaper and more reliable services of gies incommercial ISPs . Furthermore, manpower cost can also be reduced asdeployment and network support will not be required rather than establish a highspeed network where they can test it. It could have been more cost effectiveandand the research staff can focus more important endeavors such as on thethedevelopment of new network technologies a better focus of their skills .

Some areas of research done during the project are: Network Administration(DNS, Mail, Web, FTP, ProxyProxy ); Multimedia over IP (Videoconference,Videostreaming); Virtual Classroom System; Digital Content System (LibraryManagement System, Digital Library); Voice over IP; Network Management; APBioBox; and IPv6 Implementation . (See Appendix 1) . TThese are maturedtechnologies already being used widely abroad and the findings show that no nonew radical or incremental research was includedinnovations were produced bythe project .

Moreover, it was also discovered that they the project hashad no establishedmechanism way of concretely concretely monitoring or documenting what new-generation network technologies technology researcheshthe R&D wereconducted and facilitated done through the network by their partners (SeeAppendix 2).


15/32


If one of PREGINETs main thrust was increaseincreased R&D research onof newnetwork technologies or other advanced system applications or tools, theirquanti ta tive measures of success could have been the following:

Number of refereed papers published , Number of patents generated , Number of innovations developed , Number of innovations commercialized , Amount of external research grants obtained and Return on R&D investment .

Return on Investment

Overall, the approved budget for the PREGINET project was P79M (Figure 4). Dueto the unavailability of information on actual breakdown or components of theproject budget, we cannot adequately assess whether the budget spending wascost-efficient.

Nevertheless, as we have earlier identified the projects deficiency in problemidentification and solutions-matching, we have established the fact that theproject budget could have been spent more efficiently and effectively if it wasused for the development of new products that can be commercialized instead of funding the infrastructure set-up, internet connection and network support topartners.


16/32


Figure 4 Project Budget Approved, Released, Spent

Overall, the funding for the PREGINET project was a total of P79,010,719.60.Although the main advantage of having a dedicated network should be that it hasa high-speed connectivity (more faster than that of commercial ISPsnormalinternet service) in order connect Philippines universities and other researchinstitutions to the global research and education networks. Consequently, thisnetwork connection should allow the creation of emerging network technologies

(that need higher bandwidth for testing).Moreover, the fact that the projects original design did not include strategies forsustainability, it is evident that the proponent did not intend to generaterevenues or recover the cost incurred in the project implementation. A CostBenefit Analysis (CBA) should have been conducted to determine whether theproject benefits adequately justifies the cost and the Return on Investment (ROI)should have been determined to assess whether it is a sound business proposalfor the government to invest in.

Obviously,

However, in terms of Return on Investment, Tthe PREGINET has not beensuccessful in proving that this project was un able to come up with anything thatwould constitute agenerate revenues for the government except the minimalservice charges they recently introduced. Furthermore, these charges cannotadequately cover its operational expenses since at present; PREGINET has beensubsidized and included as part of ASTIs annual budget. Return of Investment.If the focus was more on the development of new network technologies andproducts that can be commercialized, it could have been the source of revenuesor returns on the governments almost P79M investment. In fact, at presentlyDOST-ASTI is spending its yearly budget just to keep the network operational.newnetwork technologies that can justify the cost. A Cost Benefit analysis betweendeveloping a dedicated network and commericialcommercial ISP shcould havebeen done. An ROI study should have been done to see the returns of this projectto its intended reciepientsrecipients. During the 5 year project, the budgetcovered the following areas: Personnel, Maintenance and Operation Expensesand Capital Outlay.


17/32


Staffing

With its original 11-man project team (Figure 5), PREGINET experienced high staff turn-over rate during the implementation. As a result, some activities were eitherdiscontinued or had to be redone/redesigned (due to lack of proper technicaldocumentation). One example of the activity that was discontinued was thedeployment of the VClass System. Initially, a number of partners becameinterested in the online learning management system. The project was able tocarry out a few trainings for institutions that expressed their interest to use itsuch as the Philippine Rice Research Institute (PhilRice) and the Benguet StateUniversity (BSU). However, when the whole development team resigned, therewas nobody left to continue with its development and deployment. The samecase happened with the development and deployment of the Greenstone DigitalLibrary, and IPv6.

Figure 5 Project Team Composition

The project experienced high staff turn-over for the duration of itsimplementation. As a result, some activities were either discontinued or had to beredone/redesigned (due to lack of proper technical documentation). One exampleof the activity that was discontinued was the deployment of the VClass System.

Total No. of Members


18/32


Initially, a number of partners became interested in the online learningmanagement system. The project was able to carry out a few trainings forinstitutions that expressed their interest to use it such as the Philippine RiceResearch Institute (PhilRice) and the Benguet State University (BSU). However,when the whole development team resigned, there was nobody left to continuewith its development and deployment. The same case happened with thedevelopment and deployment of the Greenstone Digital Library, and IPv6.

One factor that can be attributed to this was the unattractive compensation andbenefits package offered to the researchers, as DOST, a line agency is covered bythe Salary Standardization Law (SSL)low salary coverage of ASTI for itsresearchers . In addition, it is a fact that competitive packages are available fromthe private sector, which realizes the real value of these skilled human resources.In this case, high staff turn-over rateTheir transfer to higher salary work becameis inevitable ; possible due to the Salary Standardization Law which could hinderASTI from addressing this .

Although there are also other motivati ng factors that ons keep researchers loyalto their employeran organization like : challenging, interesting & important projectassignments ;, adequate research facilities and , personnel, equipment & time,and , status that maintain pP raise and recognition from peers in the scientificcommunity , the attractiveness and value of these may vary from employee toemployee . One factor that this project was not able to address sufficiently wasthe responsive and equitable reward & recognition system which can also coversalary, bonuses, monetary benefits or non monetary benefits like peerrecognition,

In any case, a system atic and establishedized documentation process could havesolvepossibly reduce the impact of d the problem disrupted and discontinued of abandoning or recreating programsprograms caused by and the lik because of staff resignations. Efficient Sdocumentation processes and knowledge transfermechanisms and protocols trict documentation should be properly in place.andknowledge transfer policies within the project should have been part of theStandard Operation Process. e.

Risk Management

As the number of connected partners increased, so did the need for a higherbandwidth as well as a more reliable network. Initially, the regional access pointswere mainly the DOTC Telecommunications Office (Telof) facilities. However,during network problems, the Telofs response time to address the problem waslimited by factors such as: lack of capacity to properly address/fix the problem;and government standard operating procedures. For example, if the technicalproblem involved an equipment that needed to be fixed/replaced/procured, the


19/32


request had to go through the usual government procedures that caused majordelays in the operation of the network.

These kinds of Risk should have been included in making contingency plans inthe beginning of the project. As a result, slow network connection, major delaysin operation and unsatisfactory service affected PREGINETs partners and client.

Product Development

Successful R&D projects, whether incremental or radical ones,ones improve sproducts & services thus contributesservices thus contribute to higherproductivity, lower costs & increased profits. It should provide customers (in thecase of PREGINET, its beneficiaries in government, academe and privateindustries ) with varied choices, better choices product performance & lowerprices. The ultimate objective of the project should have been Advanced networkservices may be are necessary to createthe development of highly advanced, first-in -the -market, cutting - edge, support next - generation applications that iscompetitive in performance and price. (especially those that need high speedinternet) which is why PREGINET was created in the first place.These In turn, newproducts c an beould have been commercialized and could possibly advance ICTdevelopment in theor make the Philippines a leader in Network Technologydevelopment . However, As mentioned earlier, researchpresent endeavors of areas covered by PREGINET , which in These new research areas iincludecludeamong others, d IpP v6, Multimedia over IP, Network Measurements, Content andApplications development, hosting and distribution , which are hardly cuttingedge already mature technologies and readily available in the first in the market .

However, tT he fact that the project did not result in a service or portfolio of products that are commercialized or new and even proven to be effectivealternatives to existing technologies makes it apparent that the project failed inthe area of product development, which should have been the foremost indicatorof the projects performance., reveals a weakness in implementation ortechnology diffusion since it should have resulted in the use of advance networktechnologies by the Government, Education and Private firms in this country..


20/32


Post- Implementation Evaluation Stage

n impact assessment 3 is a process aimed at structuring and supporting the development of policies. It identifies and assesses the problem at stake and the objectives pursued. It

identifies the main options for achieving the objective and analyses their likely impacts in the economic, environmental and social fields. It outlinesadvantages and disadvantages of each option and examines possiblesynergies and trade-offs."

TAThe Philippine Council for Advanced Science and Technology Research andDevelopment (PCASTRD) under DOST was tasked to monitor the project

accomplishments for the most part of project implementation based on thereports submitted by the project team in compliance with DOST policies andprocedures. On the other hand, the Development Academy of the PhilippinesCenter for Knowledge Management (DAP-CKM) was commissioned by ASTI toconduct the impact assessment of the PREGINET Project. Thus, in this section,we will evaluate the post-evaluation stage of the project in view of the impactassessment conducted by DAP and the analysis will be focused on the followingaspects:

Evaluation Team Evaluation Objectives

Evaluation Methods Findings/ Results of DAP Evaluation DAP Recommendations

Evaluation Team

The DAP project team consisted of Elena Avedillo-Cruz, Managing Director,Cristina Bobis, Project Officer, Sheryl dela Cruz, Technical and Administrative

CC


21/32


Staff, Caridad Aspiras, Supervising Fellow, and Segundo E. Romero, TechnicalConsultant.

Impact Assessment, http://en.wikipedia.org/wiki/Impact_assessment

The result of the impact evaluation could have been considered objectivebecause it was conducted by a third party organization. However, in the set-up,ASTI was the contracting party , project proponent and implementer. In order toensure the impartiality of the project evaluation, the contracting party to theagreement with the evaluating team should be independent from that of theproject proponent or implementing unit. As the funding and monitoringagencies for the project, DOST or PCASTRD, respectively could have been thecontracting party for the evaluation.

Evaluation Objectives

In terms of impact, ASTI agreed to the following DAP evaluation objectives:

1. Determine whether the stated project goals and objectives were achieved;

2. Ascertain the contribution of the project to the overall program objectives;

3. Assess the benefits of the projects; 4. Determine conditions under which the projects are most effective;

5. Identify the factors that contributed to the projects success or failure; 6. Recommend possible projects that can be pursued in this field and

suggestions to improve performance of such projects included.

Just like the objectives set by the PREGINET project during the proposal stage,the above objectives are vague and should have also been patterned after theSMART objectives. Moreover, assessing the impact of the project in terms of R&D can be most appropriately expressed through quantifiable indicators suchas the number of new network technologies developed.

Evaluation Methods

There were two major methods used in evaluating the projects impact thedesk review and the field assessment. For the desk review, relevant andexisting project documents were analyzed and the preliminary evaluationwhich resulted from the desk review was then validated using three sets of evaluation tools the survey questionnaire, the key informant interviews (KII)and the case studies.
http://en.wikipedia.org/wiki/Impact_assessmenthttp://en.wikipedia.org/wiki/Impact_assessment


22/32


However, it was noted that only 29 (37%) of the 78 target respondentsaccomplished and returned the forms to DAP for data processing, which wereused as basis for the evaluation (Figure 6). This shows that the basis for theevaluation is not representative of the total population rendering the results tobe dubious and questionable.

Figure 6 Retrieval Rate of Survey QuestionnaireDAP Findings

DAP evaluated the project in terms of four major areas project design,delivery, impact, and sustainability. According to DAP:

The project objectives were attained because it was able to:

a.

advance the development of next generation technologies,applications and services and made these accessible for user communities;b.provide a platform that enabled the introduction,development, testing and deployment of new network technologies; and c.build up its partner institutions from the academe,government, and research institutions with around 169partnerships established as of August 2008

The project remained focused on the objectives it set to achieve. Thus, it was able to establish a national REN,develop new technologies, and build the capability of humanresources to support the achievement of these objectives.

The result that the project objectives were attained is not adequatelyjustified given the fact that the accomplishments were not measurable. In


23/32


addition, if assessed in the light of advancing the development of next-generation technologies, only Pawikan was developed but was not successfulas it was not fully commercialized.

The overall assessment for PREGINETs project design is summarized below:

a.

The intent for the PREGINET project was clear from thestart, except for the quantifiable and measureable targetsthat should have been set at the start. This was confirmed by the documents reviewed as well as the results of thefield validation where the key informants consulted cited the following as bases for identifying the services, productsand technologies promoted by ASTI:

i. Anticipated and visible need of the country in network technologies, which is usually not pushed or probed by the private sector

ii. Need for the country to have an R&E network (REN) to be able to

iii. Participate or have access with the research activities of other RENs around the world

iv. Needs and requirements of the user communities of PREGINET in line with the thematic areas beingfocused by ASTI, namely environment, education,government, and enterprise development

v. Focus of PREGINET is the development and adoption of advanced networking technologies that can benefit PREGINET partners.

vi. Services provided are identified based on the technologies and applications used to promote thedevelopment of advanced networks.

DAPs overall assessment that the project was successful is contradicted bytheir own finding that there were no quantifiable and measureable targets setat the start of the project. It will not be possible to fully assess and effectivelyevaluate a projects impact without benchmarking it with SMART objectives.Therefore, PREGINETs success as declared by DAP may be disputed.

b.

Guided by the strategic objective to establish anationwide broadband network that will link research and education institutions for the development, testing, and deployment of next generation network technologies,


24/32


services, and applications intended for distance education,agriculture, disaster mitigation, telemedicine, and bioinformatics, the PREGINET project employed strategiesthat were found to be responsive to identified needs.These areas are as follows: a) acceleration of government connectivity, b) development of IT manpower base, c)promotion of technological innovation and experimentation, and d) creation of strategic partnershipsamong local institutions. The project, however, could havebeen optimized had it been promoted extensively. It seemed that the infrastructure was already provided by the project but content development was inadequate. Inaddition, the projects original design did not includestrategies for sustainability. It was during the latter part of the project that sustainability measures and mechanismswere incorporated into the design.

Another DAP finding is that the PREGINET project employed strategies thatwere found to be responsive to identified needs. This is also contrary to factthat there was no concrete proof that the project led to the acceleration of government connectivity, development of IT manpower base, promotion of technological innovation and experimentation, and creation of strategicpartnerships among local institutions.

DAP Recommendations

Based on the results of the evaluation, DAP recommended the following:

a. Strengthen advocacy activities to promote the benefits of the PREGINET. In spite of the attainment of project objectivesthere is still a lingering perception that the partners have not fully realized or appreciated the benefits of the network.b. Actively solicit content from the partners. While there is an existing infrastructure, the lack of content to offer to partnersshould be addressed c. Seek the involvement of other stakeholders for advocacy such as the executive and legislative branches of thegovernment. The PREGINET project has been funded from thenational government budget so it should be supported by therest of the government sector to optimize its usefulness.d. PREGINET should speak for itself. It should publicize its accomplishments in the form of stories, case studies, and best practices.e. There is no substitute for better service. Partners who were highly satisfied with PREGINET's technologies, applications


25/32


and services could give testimonials to inform and inviteothers to the network.

If DAP concentrated on assessing the project in terms of measurableobjectives, specifically the development of new technologies or products, it isapparent that the project was not successful. In view of this, we recommendthe following for future DOST projects:

a. Objective s should be s pecific, m easurable, a ttainable, relevant and t ime- bound (SMART).

b. Tools for f inancial a nalysis like CBA and financial indicators such as ROI, NPV should be used in project prioritization and selection. However, thisdoes not mean that financial tools and indicators are the only criteria forproject selection because a well-balanced project portfolio requiresinclusion of a few radical projects that would facilitate the long-termfuture viability of the organization.

c. Quantitative measures should be set to determine the projects success, such as:

- Number of refereed papers published - Number of patents generated - Number of innovations developed - Number of innovations commercialized - Amount of external research grants obtained - Return on R&D investment

d. If competitive compensation and benefits package cannot be granted to skilled research staff, provide challenging, interesting & importantproject assignments; adequate research facilities and equipment, amongothers and put in place efficient documentation processes andknowledge transfer mechanisms and protocols to reduce the projectdiscontinuity caused by staff turn-over.

e. Maintain the objectivity of the project evaluation by ensuring that the contracting party to the agreement with the evaluating team isindependent from that of the project proponent or implementing unit.

f. Monitoring processes, tools and mechanisms should be established and project evaluation should not be done only after project completion.Instead, for long-term projects, annual project audit and monitoring of accomplishments should be conducted to ensure that for each projectphase, the project milestones and targets are achieved.

he Advanced Science and Technology Institute (ASTI) of the Department of Science and Technology (DOST) commissioned the Development Academy of the Philippines Center for Knowledge Management (DAP-CKM) to conduct animpact evaluation of the VCTI-IT: Development of a High PerformanceResearch and Education Network: The Philippine Research, Education and


26/32


Government Information NETwork (PREGINET),which is a component of theCOMPETE program funded under DOSTs Grants-In-Aid (GIA).

DAP Evaluation Team and Time Frame

The DAP project team consisted of Elena Avedillo-Cruz, Managing Director,Cristina Bobis, Project Officer, Sheryl dela Cruz, Technical and AdministrativeStaff, Caridad Aspiras, Supervising Fellow, and Segundo E. Romero, TechnicalConsultant.

The Philippine Council for Advanced Science and Technology Research andDevelopment (PCASTRD) handled the monitoring of PREGINET and VCTI-RFprojects since information technology and microelectronics were the priorityareas of PCASTRD. However, PCASTRD only took actual charge of the projectafter the first implementation year.

The result of the impact evaluation could have been considered objectivebecause it was conducted by a third party organization. However, in the set-up,ASTI was the contracting party and at the same time the implementing agencyof PREGINET. As the funding and monitoring agencies respectively, DOST orPCASTRD should have been the contracting party to ensure the impartiality of the assessment.

Evaluation Objectives

In terms of assessing impact, the evaluation aimed to:

1. Determine whether the stated project goals and objectives were achieved;2. Ascertain the contribution of the projects to the overall program objectives;3. Assess the benefits of the projects;4. Determine conditions under which the projects are most effective;5. Identify the factors that contributed to the projects success or failure;6. Recommend possible projects that can be pursued in this field and

suggestions to improve performance of such projects included.

Evaluation Methods

There were two major methods used in evaluating the projects impact the deskreview and the field assessment. For the desk review, relevant and existing projectand project-related documents were analyzed mainly to reconstruct the projectdesign and to track the project results and accomplishments. The preliminaryevaluation which resulted from the desk review was then validated

DAP Findings


27/32


DAP evaluated the project in terms of four major areas project design, delivery,impact, and sustainability. According to DAP:

The project objectives were attained because it was able to:d. advance the development of next generation technologies, applicationsand services and made these accessible for user communities;e. provide a platform that enabled the introduction, development, testing and

deployment of new network technologies; andf. build up its partner institutions from the academe, government, and

research institutions with around 169 partnerships established as of August 2008.

It remained focused on the objectives it set to achieve. Thus, it was able toestablish a national REN, develop new technologies, and build the capabilityof human resources to support the achievement of these objectives.

Some of the factors considered helpful to the project included:a. International recognition and support that enabled the country to become

integrated with the global research communityb. Strengthened partnership with international research and education

networks that enhanced connectivity to the research and educationcommunity

c. Continuing R&D activities to keep abreast with current trends in advancednetwork technologies

d. Dynamic and young staff who were flexible, open to learning, and willing tospend extra hours to deliver the project

e. Support of DOSTf. Advocacy on the benefits of the project

However, certain conditions should be improved such as optimizing someinternal systems, e.g. after-sales service, billing system, documentation of processes, and policies for the use of PREGINET services.

The overall assessment for PREGINETs project design is summarizedbelow.a. The intent for the PREGINET project was clear from the start, except for the

quantifiable and measureable targets that should have been set at thestart. This was confirmed by the documents reviewed as well as the results

of the field validation where the key informants consulted cited thefollowing as bases for identifying the services, products and technologiespromoted by ASTI:

vii. Anticipated and visible need of the country in network technologies,which is usually not pushed or probed by the private sector

viii. Need for the country to have an R&E network (REN) to be able toix. Participate or have access with the research activities of other RENs

around the world


28/32


x. Needs and requirements of the user communities of PREGINET in linewith the thematic areas being focused by ASTI, namely environment,education, government, and enterprise development

xi. Focus of PREGINET is the development and adoption of advanced

networking technologies that can benefit PREGINET partners.xii. Services provided are identified based on the technologies and

applications used to promote the development of advanced networks.

b. Guided by the strategic objective to establish a nationwide broadbandnetwork that will link research and education institutions for thedevelopment, testing, and deployment of next generation networktechnologies, services, and applications intended for distanceeducation, agriculture, disaster mitigation, telemedicine, andbioinformatics, the PREGINET project employed strategies that werefound to be responsive to identified needs. These areas are as follows:

a) acceleration of government connectivity, b) development of ITmanpower base, c) promotion of technological innovation andexperimentation, and d) creation of strategic partnerships among localinstitutions. The project, however, could have been optimized had itbeen promoted extensively. It seemed that the infrastructure wasalready provided by the project but content development wasinadequate. In addition, the projects original design did not includestrategies for sustainability. It was during the latter part of the projectthat sustainability measures and mechanisms were incorporated intothe design.

DAP Recommendations

f. Strengthen advocacy activities to promote the benefits of thePREGINET. In spite of the attainment of project objectives there is still alingering perception that the partners have not fully realized orappreciated the benefits of the network.g. Actively solicit content from the partners. While there is an existinginfrastructure, the lack of content to offer to partners should beaddressedh. Seek the involvement of other stakeholders for advocacy such asthe executive and legislative branches of the government. The PREGINET

project has been funded from the national government budget so itshould be supported by the rest of the government sector to optimize itsusefulness.i. PREGINET should speak for itself. It should publicize itsaccomplishments in the form of stories, case studies, and best practices.j. There is no substitute for better service. Partners who were highlysatisfied with PREGINET's technologies, applications and services couldgive testimonials to inform and invite others to the network.


29/32


ANALYSIS

How is the performance of the network?Did DAP conduct an interview with the users of the REN?How was the service offered to its partners? Was it marketed aggressively?Are there LGUs using the network?What is the criteria for becoming a partner?What were the actual applications developed?But public appreciation for the PREGINET is still somewhat low. Thus,there is really a need to promote the benefits of the network. What are theydoing about this?Further, the technical competency of the staff involved in the implementationof PREGINET is being continuously upgraded through exposure andparticipation in gatherings where application of the ICT technologies are testedand demonstrated. There is an expressed need, however, during the dataFinal Report26gathering phase to further strengthen the R&D group of PREGINET particularlyon identifying services and technologies that would be beneficial. What arethey doing about this?

Is there a target no. of institutions per year that will become a partner?No measurable objectiveWhat new technologies were developed?

However, certain conditions should be improved such as optimizing someinternalsystems, e.g. after-sales service, billing system, documentation of processes,andpolicies for the use of PREGINET services.The project has definitely opened opportunities for ICT development andadvancement in the country. Through the development and testing of applicationsin new areas, several network and information and communicationtechnologies havebeen enhanced with PREGINET.PREGINET has made its mark with the TELCOs. The growing appreciation of theTELCOs on the importance of PREGINET role is itself recognition of the benefitsthe project has brought about in helping the advancement of networktechnology


30/32


and strengthening the network infrastructure. What does this mean?

Conclusion-B

CONCLUSIONased on our analysis of the PREGINET Project, we arrived at theconclusion that the project was unsuccessful and did not have a significantimpact on R&D. This is based on thefact that if we were to evaluate the project and focus on the R&D aspect, thefollowing will be the results:

Only one network technology, a monitoring tool called Pawikan was developed but it was not commercialized renderingthe project failed in terms of R&D output.

In terms of cost-efficiency, the project failed as evident in the fact that during the 7-year project duration, there were no successful newtechnologies developed and the research activities were mere adoptionof available and matured technologies that does not justify the P79Mproject cost.

The project lacks effective monitoring tools and mechanisms to determine the R&D output of partner-beneficiaries such as researchpapers published and patents granted, if there were any.

The project did not generate enough revenues to maintain its operation s or at least recover the governments investment.

PREGINET services ha ve no competitive advantage over other

commercially-available services, as it does not provide unique features,competitive rate or better performance.

Research done over the project duration are not classified asadvanced network technologies, but rather adpatation of foreigntechnologies. It cannot be considered a success in advancing new, nextgeneration network technologies for the Philippine Government,

4


31/32


Academe and Private firms. The lack of measures and Key performanceindicators for R&D, both for PREGINET and its PARTNERS, led to nonmeasurement of R&D output, papers, research, patents. Morever, itcannot justify DOSTs return of Investment on the project in this 7 yearproject.

In relation to this, there should have been new products andservices that resulted from R&D that can benefit society or prove to be abetter alternative for Filipino users. But seeing how there was a lack of

technology diffusion, lack of commercialization and there is stillpreference for private network providers, PREGINET was not successful inthis aspect.

Low monetary benefits affect the work motivations of Researchersand thus, led to increase turn-over of staff. An offshoot of high turnoverand lack of SOP regarding documentation and knowledge transfer is thenon-deployment of some projects when researchers resign.

List of References

Appendix

Appendix 1. List of Research Areas duingduring PREGINET project

IPv6 - the standard for the next generation Internet, and which addresses the limitations of the current Internet protocol, IPv4;

Multimedia over IP - which will deploy H.323 and multimedia (voice and video) streaming as standard network services for the research and academiccommunity to provide distance learning, telemedicine, and other uses of videoconferencing and multimedia streaming;

5


32/32


Network Measurements - which involve the use of network monitoring tools and systems to measure the performance of various services and applicationsunder test as well as to monitor network health;

Network Infrastructure - which deals with the establishment of the PREGINET

nationwide network and the study of different wire and wireless broadbandnetwork technologies; and Content and applications development, hosting and distribution which

includes the provision of high-availability servers and computing systems forscientific applications that shall be made available for use by the partnerinstitutions for their projects over PREGINET.

Appendix 2. PREGINET Partners

Documents

TM 204 PREGINET Complete Paper + Bianca + Bong + Cha