CIWQS Review Phase II:CIWQS Review Phase II:

Evaluation and Final Evaluation and Final

RecommendationsRecommendations

March 14, 2008March 14, 2008

BackgroundBackgroundBackgroundBackground Review Panel convened in response to Review Panel convened in response to

State Water Board’s request for external State Water Board’s request for external review of CIWQSreview of CIWQS

First Panel meeting held in May 2007First Panel meeting held in May 2007

Preliminary Panel report with Preliminary Panel report with recommendations issued July 2007recommendations issued July 2007

Water Board staff to consider Water Board staff to consider recommendations and demonstrate recommendations and demonstrate progress toward benchmarks through progress toward benchmarks through February 2008February 2008

Findings of Previous Findings of Previous ReviewReview

Findings of Previous Findings of Previous ReviewReview

Seven essential recommendations:Seven essential recommendations: Reduce the project’s scopeReduce the project’s scope Restructure CIWQS’ project managementRestructure CIWQS’ project management Validate the system’s requirementsValidate the system’s requirements Rebuild key constituenciesRebuild key constituencies Address data quality issuesAddress data quality issues Produce key reportsProduce key reports Improve user interfaceImprove user interface

Goals for Phase IIGoals for Phase IIGoals for Phase IIGoals for Phase II

Water Board staff will demonstrate Water Board staff will demonstrate progress by responding to Phase I progress by responding to Phase I recommendationsrecommendations

Including specific actions, timelines, and Including specific actions, timelines, and additional performance measures for the coming additional performance measures for the coming yearyear

Panel will review progress and prepare final Panel will review progress and prepare final report to:report to:

Judge adequacy of Water Board’s responseJudge adequacy of Water Board’s response Make final recommendations for future direction Make final recommendations for future direction

of the projectof the project

Overview of Final ReviewOverview of Final ReviewOverview of Final ReviewOverview of Final Review

CIWQS remains essential for the State Water Board to CIWQS remains essential for the State Water Board to fulfill its missionfulfill its mission

Terminating funding would be counterproductive Terminating funding would be counterproductive

CIWQS should be funded at an appropriate levelCIWQS should be funded at an appropriate level

The program’s new direction should lead to successThe program’s new direction should lead to success We are impressed with the remarkable turnaround since MayWe are impressed with the remarkable turnaround since May

We are particularly impressed with progress in We are particularly impressed with progress in reorganizing management and communicating with reorganizing management and communicating with usersusers

Overview (cont.)Overview (cont.)Overview (cont.)Overview (cont.)

The redefinition of project scope was The redefinition of project scope was appropriate and effectiveappropriate and effective

We have remaining concerns with some We have remaining concerns with some aspects of the technical approach, rate of aspects of the technical approach, rate of data cleanup, and report productiondata cleanup, and report production

Original RecommendationsOriginal RecommendationsOriginal RecommendationsOriginal Recommendations

Reduce the scope Reduce the scope

Restructure management Restructure management

Validate system requirementsValidate system requirements

Rebuild constituencyRebuild constituency

Address data issuesAddress data issues

Produce key reports Produce key reports

Improve user interfaceImprove user interface

Reduce the ScopeReduce the ScopeReduce the ScopeReduce the Scope

EvaluationEvaluation Achieved significant scope reductionAchieved significant scope reduction Increased probability of project successIncreased probability of project success Achieved buy-in from Steering Committee & usersAchieved buy-in from Steering Committee & users Phased eSMR implementation plan is reasonablePhased eSMR implementation plan is reasonable Implementation through eSMR Level 3 is desirableImplementation through eSMR Level 3 is desirable We concur with need for a separate federal interfaceWe concur with need for a separate federal interface

RecommendationRecommendation Consider trimming unneeded parts of the database to Consider trimming unneeded parts of the database to

meet reduced project scopemeet reduced project scope

Restructure ManagementRestructure Management Restructure ManagementRestructure Management

EvaluationEvaluation The new management structure positions CIWQS for The new management structure positions CIWQS for

successsuccess Effective new organization with leadership in Water Quality Effective new organization with leadership in Water Quality

not in DIT ties the project closer to the Board’s missionnot in DIT ties the project closer to the Board’s mission Buy-in from senior managers for the project’s importance Buy-in from senior managers for the project’s importance

and new directionand new direction

Validate System RequirementsValidate System RequirementsValidate System RequirementsValidate System Requirements

EvaluationEvaluation Fundamentally improved systems design & implementationFundamentally improved systems design & implementation Corrected design and approach based on testingCorrected design and approach based on testing The underlying structure can be made to workThe underlying structure can be made to work

Referential integrity issues are being addressedReferential integrity issues are being addressed Requirements better reflect user inputs & needsRequirements better reflect user inputs & needs

RecommendationsRecommendations Demonstrate a disciplined systems engineering processDemonstrate a disciplined systems engineering process

Document team members’ roles, responsibilities, activitiesDocument team members’ roles, responsibilities, activities Use test cases to demonstrate successful resolution of Use test cases to demonstrate successful resolution of

referential integrity issuesreferential integrity issues

Rebuild ConstituencyRebuild ConstituencyRebuild ConstituencyRebuild Constituency

EvaluationEvaluation Achieved remarkable cultural change that significantly Achieved remarkable cultural change that significantly

improved user communication and supportimproved user communication and support Developed effective user community structureDeveloped effective user community structure Users attended Panel meeting and strongly supported the Users attended Panel meeting and strongly supported the

projectproject Users strongly engaged in most areasUsers strongly engaged in most areas Issues with stormwater group illustrate cultural Issues with stormwater group illustrate cultural

improvements have further to goimprovements have further to go

RecommendationsRecommendations Should apply approach used for eSMR to all groups and Should apply approach used for eSMR to all groups and

issuesissues

Address Data IssuesAddress Data IssuesAddress Data IssuesAddress Data Issues EvaluationEvaluation

Less was accomplished than we had expectedLess was accomplished than we had expected Acknowledged the need for systematic QA/QCAcknowledged the need for systematic QA/QC Database integrity is being addressedDatabase integrity is being addressed Some progress on correcting existing data errorsSome progress on correcting existing data errors

RecommendationsRecommendations Devote more focused effort to QA/QC and data cleanupDevote more focused effort to QA/QC and data cleanup Develop coherent plan that prioritizes data cleanup and Develop coherent plan that prioritizes data cleanup and

identifies needed resources and timeframeidentifies needed resources and timeframe Create formal QA/QC mechanisms, including user Create formal QA/QC mechanisms, including user

participationparticipation Make QA/QC an integral part of database administration Make QA/QC an integral part of database administration

and operationand operation

Produce Key ReportsProduce Key ReportsProduce Key ReportsProduce Key Reports

Evaluation Evaluation Created a mechanism to identify and prioritize key reportsCreated a mechanism to identify and prioritize key reports Made some progress on developing canned reportsMade some progress on developing canned reports

RecommendationsRecommendations Demonstrate that required reports can be produced from Demonstrate that required reports can be produced from

the core CIWQS systemthe core CIWQS system Increase staff’s ability to create ad hoc reportsIncrease staff’s ability to create ad hoc reports Accelerate report productionAccelerate report production

Improve User InterfacesImprove User InterfacesImprove User InterfacesImprove User Interfaces

EvaluationEvaluation Created a mechanism for input and feedback from usersCreated a mechanism for input and feedback from users Created a successful and easy-to-use SSO moduleCreated a successful and easy-to-use SSO module Seem to be heading in the right directionSeem to be heading in the right direction

RecommendationsRecommendations Create a formal mechanism for validating and testing user Create a formal mechanism for validating and testing user

interfacesinterfaces Create context-specific drop-down menusCreate context-specific drop-down menus Create context-specific online helpCreate context-specific online help

Additional RecommendationsAdditional RecommendationsAdditional RecommendationsAdditional Recommendations

Build a non-CIWQS interface with ICIS-NPDESBuild a non-CIWQS interface with ICIS-NPDES

Develop a set of persistent end-to-end tests to Develop a set of persistent end-to-end tests to demonstrate that the data model and demonstrate that the data model and processes workprocesses work

Develop a deployment plan for the system and Develop a deployment plan for the system and datadata

Fund the system at an appropriate levelFund the system at an appropriate level

ICIS-NPDES Issue Must Be ResolvedICIS-NPDES Issue Must Be ResolvedICIS-NPDES Issue Must Be ResolvedICIS-NPDES Issue Must Be Resolved

ICIS-NPDES will not meet State Board’s needsICIS-NPDES will not meet State Board’s needs

There are different reporting needs for EPA and There are different reporting needs for EPA and for State Boardfor State Board

State system should cross-check violation State system should cross-check violation determinations between state and federal determinations between state and federal systems to quickly identify erroneous violationssystems to quickly identify erroneous violations

Download ICIS-NPDES data to CIWQS as needed Download ICIS-NPDES data to CIWQS as needed for State Board purposesfor State Board purposes

Evaluate Interface Options w/EPAEvaluate Interface Options w/EPAEvaluate Interface Options w/EPAEvaluate Interface Options w/EPA

Consider discharger submission usingConsider discharger submission using eDMR (file-oriented)eDMR (file-oriented) netDMR (interactive)netDMR (interactive)

Develop test plan and case(s) including EPA DMR Develop test plan and case(s) including EPA DMR data download to state systemdata download to state system

EPA has agreed to this using currently submitted data; EPA has agreed to this using currently submitted data; this should be formalized quickly to take advantage of the this should be formalized quickly to take advantage of the offeroffer

Develop End-to-End TestingDevelop End-to-End TestingDevelop End-to-End TestingDevelop End-to-End Testing

Develop and maintain a library of test Develop and maintain a library of test cases with known results that can be used cases with known results that can be used to test system and processes as changes to test system and processes as changes are madeare made

Should be implemented as part of all Should be implemented as part of all deployment plans on an ongoing basisdeployment plans on an ongoing basis

Develop Deployment PlanDevelop Deployment PlanDevelop Deployment PlanDevelop Deployment Plan

Develop a deployment plan for every Develop a deployment plan for every system update and releasesystem update and release

Should include software, hardware, data, Should include software, hardware, data, and quality control aspectsand quality control aspects

End-to-end testing should be integral part End-to-end testing should be integral part of the planof the plan

Each phase of the eSMR development Each phase of the eSMR development needs a deployment planneeds a deployment plan

Fund CIWQS at Appropriate LevelFund CIWQS at Appropriate LevelFund CIWQS at Appropriate LevelFund CIWQS at Appropriate Level

Panel is encouraged by progressPanel is encouraged by progress

Believe that changes make it likely the Believe that changes make it likely the CIWQS project will result in a system that CIWQS project will result in a system that serves the State Board’s needs for serves the State Board’s needs for accuracy, automation, and efficiency for accuracy, automation, and efficiency for the futurethe future

Encourage State Board and Legislature to Encourage State Board and Legislature to fund the development at levels that can fund the development at levels that can maintain and accelerate developmentmaintain and accelerate development

Next StepsNext StepsNext StepsNext Steps

Full report of findings and recommendations, with Full report of findings and recommendations, with performance metrics, within 2 monthsperformance metrics, within 2 months

Panel open to revisiting progress in 9 – 12 months if Panel open to revisiting progress in 9 – 12 months if askedasked

SummarySummarySummarySummary

The project has made substantial improvements and The project has made substantial improvements and is on a path to successis on a path to success

Management restructuring, rebuilding user constituencies, Management restructuring, rebuilding user constituencies, and system validation are clear successesand system validation are clear successes

Additional progress needed on data cleanup, reporting, user Additional progress needed on data cleanup, reporting, user interfacesinterfaces

Additional recommendations include:Additional recommendations include: Build a non-CIWQS interface with ICIS-NPDESBuild a non-CIWQS interface with ICIS-NPDES Develop end-to-end tests to demonstrate all functionalityDevelop end-to-end tests to demonstrate all functionality Develop comprehensive deployment planDevelop comprehensive deployment plan Fund CIWQS at an appropriate levelFund CIWQS at an appropriate level