General IT Cotnrols Review of the Division of · PDF fileCity of Philadelphia Division of Technology 1234 Market Street, Suite 1850 ... Susan LaCour Technology Manager BASIS2 Security

General IT Controls Review of the Division of Technology

Fiscal 2008

February 18, 2009 Mr. Allan R. Frank, Chief Information Officer City of Philadelphia Division of Technology 1234 Market Street, Suite 1850 Philadelphia, PA 19107-3721 A review of the Division of Technology’s information systems controls was performed with the assistance of SMART Business Advisory and Consulting, LLC. The purpose of the engagement was to evaluate the effectiveness of the Division of Technology’s information systems control structure as it applies to the general information technology (IT) controls over selected financially significant systems. A synopsis of the results of the review is provided in the executive summary of the report. The findings and recommendations contained in the report were discussed with department officials at an exit conference. We included the Division of Technology’s written response to the findings and recommendations as part of the report. We believe that, if implemented by management, these recommendations will improve the Division of Technology’s information systems controls. We would like to express our thanks to the management and staff of the Division of Technology and IT personnel within end-user departments for the courtesy and cooperation shown during the conduct of this review. Very truly yours, ALAN BUTKOVITZ City Controller cc: Honorable Michael A. Nutter, Mayor Honorable Anna C. Verna, President and Honorable Members of City Council Camille Barnett, Ph.D., Managing Director and Other Members of the Mayor’s Cabinet

General IT Controls Review of the Division of Technology For the Period of July 1, 2007 through June 30, 2008

Prepared for: City of Philadelphia, Office of the Controller January 15, 2009 www.smartgrp.com

January 31, 2009 Mr. Alan Butkovitz City Controller City of Philadelphia Office of the Controller 1230 Municipal Services Building 1401 John F. Kennedy Blvd. Philadelphia, PA 19102-1670 Dear Mr. Butkovitz, We have reviewed the general controls of the information technology (“IT”) functions of the City of Philadelphia’s Division of Technology. The objective of our review was to evaluate the effectiveness of the Division of Technology information systems control structure as it applies to the general IT controls environment of selected key financial systems. We concentrated our review on the organization environment, system access and security, application development, system software, processing, and disaster planning and contingency process controls as part of the Office of the Controller’s City of Philadelphia 2008 Fiscal Year Financial Statement audit. Our fieldwork was performed from October 20, 2008 through January 15, 2009. Our observations, related risks, recommendations, and management’s responses to our observations are included in this report. SMART discussed our observations and recommendations with the Division of Technology during an exit conference on January 31, 2009. We would like to thank the City of Philadelphia’s Office of the Controller for the opportunity to assist you and your staff in the performance of this review of the Division of Technology general IT controls. Kind Regards,

SMART Business Advisory and Consulting, LLC.

City of Philadelphia, Office of the Controller General IT Controls Review of the Division of Technology

For the Period of July 1, 2007 through June 30, 2008

TABLE OF CONTENTS

PROJECT TEAM AND CITY OF PHILADELPHIA IT CONTACTS.................................................. 1 EXECUTIVE SUMMARY .......................................................................................................................... 3

PURPOSE ...................................................................................................................................................... 3 BACKGROUND INFORMATION ...................................................................................................................... 3 SCOPE AND APPROACH ................................................................................................................................ 3 KEY OBSERVATIONS .................................................................................................................................... 4 CONCLUSION................................................................................................................................................ 5

DETAILED FINDINGS AND PROCESS IMPROVEMENT OPPORTUNITIES................................ 6 FINDINGS ..................................................................................................................................................... 6

IT Governance.......................................................................................................................................... 6 Access to Programs and Data.................................................................................................................. 7 Change Management ............................................................................................................................. 12 Computer Operations............................................................................................................................. 14

OPPORTUNITY FOR PROCESS IMPROVEMENT.............................................................................................. 15 APPENDIX A: SUPPLEMENTAL BACKGROUND INFORMATION............................................. 16

OVERVIEW ................................................................................................................................................. 16 ORGANIZATION ENVIRONMENT CONTROLS (IT GOVERNANCE)................................................................. 16 SYSTEM ACCESS AND SECURITY CONTROLS (ACCESS TO PROGRAMS AND DATA) .................................... 17

FAMIS and FAMIS Subsystems.............................................................................................................. 18 Other Mainframe Systems ...................................................................................................................... 18 BASIS2 Water Billing System................................................................................................................. 19

APPLICATION DEVELOPMENT AND SYSTEM SOFTWARE CONTROLS (CHANGE MANAGEMENT) ................. 19 PROCESSING, DISASTER PLANNING, AND CONTINGENCY CONTROLS (COMPUTER OPERATIONS) .............. 20

Physical Security over the Data Center ................................................................................................. 20 Environmental Controls over the Data Center ...................................................................................... 20 Backup Procedures ................................................................................................................................ 21 Disaster Planning and Contingency ...................................................................................................... 21

City of Philadelphia, Office of the Controller General Controls Review of the Division of Technology


Page 1 of 21

PROJECT TEAM AND CITY OF PHILADELPHIA IT CONTACTS

Name Title Project Role Contact Information

SMART

John McLaughlin Senior Managing Director Quality Assurance [email protected]

John Cutrona Senior Manager Project Manager [email protected]

Derek Danilson Manager Project Lead [email protected]

Stephen Logan Senior Consultant Project Staff [email protected]

Steven Penzimer Consultant Project Staff [email protected]

City of Philadelphia IT Contacts

Allan Frank Chief Information Officer Project Contact [email protected]

Jim White Delivery Support Services Project Contact [email protected]

Connie Talbert Administrative Assistant Audit Request Facilitator [email protected]

Michael King Deputy CIO Security Officer [email protected]

Lisa Coleman Enterprise Manager Mainframe Security, Disaster Recovery

[email protected]

Howell Herring Deputy Director of IT Mainframe Security [email protected]

Derek Alston Sys. Program Supervisor FAMIS Security [email protected]

Kathleen Beckman Supervisor FAMIS Security [email protected]

John Corlies Network Administrator Finance, FAMIS Security [email protected]

Michael Kauffman Deputy Finance Director Finance, FAMIS Security [email protected]

Daniel Smith Sr. IS Development Manager ADABAS Security [email protected]

Susan LaCour Technology Manager BASIS2 Security [email protected]

Regina Post Open Sys. Administrator BASIS2 Security [email protected]

Jeffrey Gardosh Info. Security Analyst Firewall Security [email protected]

Robert Conzelman I.S. Project Manager Change Management [email protected]

Benjamin Feld Systems Program Project Supervisor

FAMIS Change Management [email protected]



Page 2 of 21

Name Title Project Role Contact Information

Gerard Grover I.S. Group Manager TIPS Change Management [email protected]

Tom D. Fenner Director of Special Project BASIS2 Change Management [email protected]

Roy Zatcoff Program Director BASIS2 Change Management [email protected]

Charles Mouteng Database Administrator Computer Processing [email protected]



Page 3 of 21

EXECUTIVE SUMMARY

Purpose

The purpose of this review was to evaluate the effectiveness of the Division of Technology (“DOT”) information systems control structure as it applies to the general IT controls environment of selected key financial systems. We concentrated our review on the organization environment, system access and security, application development, system software, processing, and disaster planning and contingency process controls as part of the Office of the Controller’s City of Philadelphia (“the City”) 2008 Fiscal Year Financial Statement audit. Our fieldwork was performed from October 20, 2008 through January 15, 2009. Our observations, related risks, recommendations, and management’s responses to our observations are included in this report. SMART discussed our observations and recommendations with the DOT during an exit conference on January 31, 2009.

Background Information

The DOT is one of at least 26 different departments within the City of Philadelphia responsible for Information Technology as identified in our interviews with management. The DOT is responsible for the City’s network infrastructure located in multiple data centers and supports over 100 of the City’s estimated 400 production applications. Of these applications, the following systems were identified as the City’s key financial systems:

• Financial Accounting Management Information Systems (“FAMIS”) • FAMIS subsystems (including ADPICS, the City’s purchasing system) • Payroll • Pension Payroll • Taxpayer Inquiry & Payment System (“TIPS”) – Revenue Accounting • TIPS – Revenue Settlement / Cashiering • Health & Welfare • BASIS2 - the City’s water billing system

All of these systems reside on the mainframe except for BASIS2 which is installed in a Microsoft Windows environment. We have included supplemental background information in Appendix A for the general IT controls areas included within the scope of this review. Referring to the details provided in this appendix will be useful in understanding the context of our observations and the variety of processes managed by IT and the level of dependencies between IT groups.

Scope and Approach

SMART obtained the inventory of applications supported by the DOT. This listing of 374 applications was reviewed by SMART, the City Controller’s Office, and the DOT to identify DOT applications that support the City’s financial statement reporting process. The review of this listing resulted in the identification of 34 DOT applications involved in the City’s financial statement reporting process. The City Controller’s Office, with support from SMART, identified seven financially significant systems out of the 34 financial systems to be included within the scope for this review. These systems include:

• Pension Payroll System • FAMIS and related FAMIS Subsystems



Page 4 of 21

• Health & Welfare • Payroll & Payroll Interfaces • TIPS – Revenue Accounting • TIPS – Revenue Settlement / Cashiering • BASIS2 - Water Billing System

SMART identified, documented, and tested Information Technology general controls over the selected financially significant systems. SMART’s approach evaluated control activities to determine if identified controls are sufficient to achieve objectives related to the following Information Technology general control areas:

• Organization and Environment (IT Governance) • Access to Data and Programs (System Access and Security), • Program and Configuration Development, Acquisition and Change Management

(Application and System software), and • Computer Operations (including processing controls, disaster recovery, and contingency

planning).

The scope of our review was limited to the DOT. As such, an assessment of the process to administer security at the application level was not performed; however, such a review should be considered. SMART’s test procedures included meetings with appropriate IT personnel within the DOT, IT personnel within the application end-user community, inspection of documentation, and the observation or re-performance of identified control activities. SMART documented the results of our procedures in the Detailed Findings section of this report. In addition, SMART reviewed the results of our procedures during an exit meeting with key DOT management and other individuals who assisted us during the execution of our procedures.

Key Observations

Our most important observations and recommendations are summarized below. Detailed observations and recommendations appear later in the report:

• IT Governance – Formal policies and procedures for the areas included within the scope of this review were not available or were not adequately documented. The Executive Order for the Mayor’s Office of Information Systems (“MOIS”), which was superseded by the DOT Executive Order, specifies that MOIS / DOT is responsible for setting policy and procedures for all City IT organizations. With at least 26 different IT groups, there is a risk of inconsistent application of controls designed to secure City data, maintain a stabilized IT environment, and maintain the availability of systems.

• Security Administration – The DOT is not responsible for administrating application security for a number of the financially significant applications in the scope of our review. This has resulted in a segregation of duties issue between security administration and transactional responsibilities in the applications within the scope of this review. We also observed that the granting of access was not role-based to ensure users are restricted to the access they need to perform their respective jobs. These observations, combined with the lack of formal policy and procedures, create a risk that applications are not configured and access is not granted on a consistent basis according to the informal standards established by the DOT.



Page 5 of 21

• Security Monitoring – Monitoring controls have not been established to regularly evaluate the configuration, authorization, and appropriateness of access to the City’s computer resources. As a result, the DOT does not have the ability to detect instances of inappropriate access. Appropriate monitoring controls will help to ensure that access for terminated employees and contractors is removed in a timely manner. Appropriate security monitoring controls are important to a decentralized IT environment in order to provide a mechanism for the DOT to ensure compliance with the policies, procedures, and standards established by the DOT.

• Change Control – The DOT Change Control Process is not applied consistently to all production systems and requires update. While the review noted exceptions to compliance with the procedures established by the DOT across all development areas, the greatest area of risk related to change control was observed within the development and maintenance of the Health & Welfare system.

• Disaster Recovery – The Disaster Recovery Plan’s documentation does not appear to include several key elements that are critical to the successful execution of the plan across all areas should a disabling event occur. The documentation provided by the DOT does not include key elements such as an evaluation of the critical systems, acceptable downtime for each system, and key contact information for each critical system. In addition, the plan provided by the DOT did not cover all environments under their responsibility, specifically the Microsoft Windows environment. Without formalized policies and procedures in place, management runs the risk of users not following a standard process, resulting in unauthorized activities and transactions, ineffective use of resources, and most importantly, the inability to restore all DOT system resources after a disabling event occurs.

Conclusion

Based on our observations during the review and testing of controls, the above controls do not appear to be operating effectively or on a consistent basis for the environments and applications included within the scope of this review.

The results of our testing have been reviewed with DOT management prior to their inclusion in this report. Management has included their responses and action plans to address the findings and recommendations where appropriate. We appreciate the assistance provided to us by members of the DOT management and staff during the course of this review.

City of Philadelphia, Office of the Controller

General Controls Review of the Division of Technology For the Period of July 1, 2007 through June 30, 2008

Page 6 of 21

DETAILED FINDINGS AND PROCESS IMPROVEMENT OPPORTUNITIES

Findings Note: Findings represent observations related to control weaknesses relevant to the risks and objectives of this review and require corrective action in order to meet the objectives.

IT Governance

No Observation/Condition Risk / Potential Effect Recommendation Management’s Response

1 Formally documented policies and procedures for a number of areas included within the scope of this review do not exist. Specifically, System Access and Security, and Computer Processing (i.e., Operations) policies were not provided by the DOT when requested.

While we received documentation of other policies and procedures, Change Management and Disaster Planning were not adequately documented.

The Executive Order for MOIS, which was superseded by the DOT Executive Order, specifies that MOIS / DOT is responsible for setting policy and procedures for all City IT organizations.

With at least 26 different IT groups, there is a risk of inconsistent application of controls designed to secure City data, maintain a stabilized IT environment, and maintain the availability of systems.

We recommend the span of control of the Division of Technology be evaluated in the context of overall governance of City-wide IT resources. Such governance should address the need for formally documented Policy and Procedures that span all IT functions in a centralized manner to set minimum standards and guidelines in which IT professionals can operate across all City Departments.

Once policies and procedures have been formally documented, the DOT should establish training and awareness programs in order to ensure that users understand their roles and responsibilities as it relates to these policies and procedures.

Finally, the DOT should design and implement appropriate monitoring controls to ensure these policies and procedures are complied with on a consistent basis.

We agree with this observation and recommendations. Efforts are currently in process to implement City-wide Information Security policies to serve as the basis for further policy creation. Additionally, efforts are under way to integrate IT functions across the City, resulting in more efficient governance. As efforts are moved forward, appropriate training and awareness programs will be implemented as well.



Page 7 of 21

Access to Programs and Data


2 Security standards are not formally documented. Documented security standards should include, but not be limited to:

• Firewall configuration • Password parameters • Anti-virus configuration • VPN encryption • Account lockout settings

A lack of documented security standards could result in inconsistent security configuration settings across the City’s IT environments that could result in unauthorized access that compromises the availability, confidentiality, and integrity of the City’s financial information.

Security standards need to be established, clearly documented, and communicated to all City personnel responsible for the implementation and maintenance of these standards.

We agree with this recommendation. Efforts are in process to establish and monitor security standards on systems across the City. Much of this will be achieved through the implementation of new systems to replace aging infrastructure.

3 Monitoring procedures have not been implemented to ensure full compliance with applicable standards. For instance, monitoring procedures should include at a minimum:

• User and administrative access reviews

• Terminated employee and contractor access reviews

• Unauthorized access attempt log reviews

• Administrator activity reviews

A lack of monitoring procedures could result in users not following a standard process, resulting in unauthorized activities and transactions and a risk to the integrity of financial data.

Monitoring procedures need to be developed to ensure compliance with the standards.

We agree with this recommendation. Monitoring is a key component of the efforts currently in process.



Page 8 of 21


4 An inadequate segregation of duties exists within five of the financially significant mainframe applications, including:

• FAMIS • Payroll • Pension Payroll • TIPS • Health & Welfare

Specifically, users who have the ability to authorize/administer user access/security over the application also have the ability to authorize and process transactions within the system. This issue may have been exacerbated by the lack of a defined governance structure over City-wide technology and incomplete policies and procedures.

Inappropriate segregation of duties could compromise the integrity of the financial data within the application.

We recommend that policies and procedures related to administration of security across all applications, databases and operating systems be implemented to ensure an appropriate segregation of duties is achieved. Specifically, the ability to add, change, and remove users from an application should be delegated to an area that is independent of the department utilizing the application or user access should be periodically monitored by personnel independent of the security administration function. We also recommend that a periodic (e.g., annual) review of user access should be completed to ensure that access is appropriately segregated and access granted to users is appropriate based on their assigned job.

DOT concurs with recommendation. The rights to add, change, and remove users will be segregated from the department utilizing the application.

5 Granting of access is not role-based to ensure user access is restricted to the access they need to perform their job. We observed that user profiles are “copied” in order to grant a new user access rather than clearly defining and restricting user access based on needed application functions and data. We specifically observed this in user access requests for FAMIS.

Unauthorized access could compromise the integrity of the financial data within the application

We recommend that management consider a role-based security project for key financial systems designed to align security rights with job titles. This project should be initiated to appropriately restrict access and create the ability for the City to consistently grant access based on job requirements. This project will require coordination between the DOT and representatives from the appropriate end user departments.

We agree with this recommendation.



Page 9 of 21


6 Terminated employees and contractors are not communicated to the appropriate individuals responsible for removing access to computer resources in a timely manner. Because security administration is decentralized and is not integrated with the personnel termination process performed by Human Resources (“HR”), user IDs remain active for terminated employees and contractors. For example, several terminated employees and contractors have active user IDs to one or more of the systems included within the scope of this review.

Unauthorized access to data by a terminated employee or contractor could compromise the integrity of the financial data within the application

We recommend a formal policy and procedure be implemented over the removal of terminated user access from all systems in which they have access. Such policies and procedures should be coordinated with the Department of Technology and the Office of Human Resources.

We agree with this recommendation. This has been raised by DOT with the Personnel department. Efforts are under way within Personnel to create a standard process to govern the management of employees. This process is being coordinated with DOT.

7 One generic user ID exists within the BASIS2 water billing system. Generic user IDs are often created during system implementations but should be deleted, for security purposes, once the system is live.

Generic user accounts cause a lack of accountability within the system.

We recommend all generic user IDs be removed. Users should be assigned unique user IDs. The use of unique user IDs assigned to authorized personnel provides management a mechanism to log, monitor, and hold individuals accountable for activities performed through the use of IDs.

This generic user ID has been eliminated. Standard policy is to not allow generic IDs within systems. This policy has been communicated and will be monitored for compliance.

8 Administrative/power user access has been granted within each system included in the scope of this review to individuals who do not appear to require such access given their job responsibilities.

Inappropriate administrative access may allow employees to perform functions that are outside their job responsibility or expertise, or potentially compromise the data within the finically significant applications.

We recommend that access to power user IDs should be restricted to those individuals who need power user access in accordance with their job responsibilities. All users who do not need this access to perform their job responsibilities should have this access removed. In addition, a monitoring control should be implemented to assist management in appropriately restricting power user access to authorized individuals.

DOT will review those users with administrative access to ensure they require this type of access to perform their job responsibilities. Those who do not require administrative access will have it revoked and replaced with access in line with their responsibilities. Monitoring controls will be put in place as recommended.



Page 10 of 21


9 Three administrative users share the firewall administrative account.

By sharing an administrative ID, management does not have the ability to hold users accountable for activity performed on the system.

We recommend that system administration rights should be granted to individual user IDs. Where possible, security rights should be assigned to groups and authorized users assigned to those groups. The use of unique user IDs assigned to authorized personnel provides management a mechanism to log, monitor, and hold individuals accountable for activities performed through the use of IDs with administrative capabilities.

By February 1, 2009 all accounts in the City of Philadelphia firewall will be individual user accounts and a policy will be in place to maintain access is only allowed in that manner.

10 The activities of administrators do not appear to be adequately monitored to ensure that inappropriate activities are being detected.

Insufficient monitoring controls could lead to unauthorized changes being implemented into the production environments that do not have a valid business purpose.

Management should evaluate its current monitoring controls, including logging, over the use of administrative user IDs. By implementing additional controls, management can verify activities are valid and potential fraudulent activities are detected.

We agree with this recommendation. Efforts are in process to establish and monitor security standards on systems across the City. Administrative access monitoring is included in this effort.

11 Users maintain multiple IDs in the Payroll System, BASIS2 Water Billing system, and the Health & Welfare system.

By maintaining multiple user IDs, a potential segregation of duties issue could exist across user IDs. Inappropriate segregation of duties could compromise the integrity of the financial data within the application.

We recommend each user should be assigned a unique user ID with security set that is in accordance with their job responsibilities.

DOT will review all user IDs within Payroll, Basis2, and Health & Welfare to identify those with multiple sign-ons. Those with multiple sign-ons will be investigated to ensure their level of access is set in accordance to their job responsibilities.



Page 11 of 21


12 Computer Access Request Forms did not exist for four out of five new BASIS2 users which were selected for testing. In addition, new user access requests did not have the appropriate approval authority for the following applications:

• TIPS • Payroll • Pension

Improper system access process for new users could allow inappropriate access within the affected system.

The new user process should be clearly defined and communicated to City of Philadelphia personnel. All new user access and change of user access should be documented and approved. The documentation around this process should be maintained.

We agree with this recommendation. This process is currently under review and will be enhanced to close any gaps.

13 Password parameters are not consistent with generally accepted security standards for the systems included within the scope of this review.

Weak password parameters could potentially compromise the systems’ security, and allow the breach of data, resources, and assets to occur by an intruder or hacker.

Password parameters should be enforced on all financially relevant applications and configured according to generally accepted security standards. The DOT should evaluate current password parameters such as minimum password length, password complexity, password expiration interval, and password history against these standards for each financial system.

We agree with this recommendation. Access Control standards are being reviewed for all systems to determine gaps. The City is migrating toward a single sign on capability based on a central directory, which will eliminate these inconsistencies.

14 User accounts are not consistently locked out (disabled) after multiple invalid login attempts.

An intruder or hacker could attempt a user ID and password login with no system limit.

According to generally accepted security standards, account lockout settings should be properly configured to prevent brute force attacks on financially relevant data.

We agree with this recommendation. Access Control standards are being reviewed for all systems to determine gaps. The City is migrating toward a single sign on capability based on a central directory, which will eliminate these inconsistencies.



Page 12 of 21

Change Management


15 Change management tickets for all five changes selected for the Health and Welfare system were not created as required by DOT policy.

Program changes that are not approved by an authorized user could compromise the integrity of the data within the affected systems. The program changes may not accurately reflect the IT initiatives that are being set forth by the business.

All changes that are implemented into production should have a corresponding change management ticket.

The change management policies and procedures are currently under review within DOT for both internal and City-wide change management. Upon completion, new change management policy and procedures will be implemented.

16 Nine of 30 change tickets did not contain approval by the submitter’s manager as required by DOT policy. The applications included in the nine exceptions were:

• FAMIS • Pension • Health & Welfare


According to the Change Management Policy, all change management tickets should evidence an approval by the submitter’s manager. If this process is not in place at the Department of Technology, the Change Management Policy should be evaluated and revised to reflect the process in place at the Department of Technology.


17 For 15 of 30 changes, approval was not obtained by the business unit in which the change was intended as required by DOT policy. The applications included in the 15 exceptions were:

• TIPS • Pension • Payroll • BASIS2 • Health & Welfare


Per the Change Management Policy, the submission of a change must be approved by the submitter’s immediate manager. If this process is not in place at the Department of Technology, the Change Management Policy should be evaluated and revised to reflect the process in place at the Department of Technology.




Page 13 of 21


18 For 15 of 30 changes, evidence of testing or an explanation as to why testing was not complete was not evidenced on the change management ticket as required by DOT policy. The applications included in the 15 exceptions were:

• TIPS • FAMIS • Payroll • BASIS2 • Health & Welfare

Changes which are not tested may cause the system to malfunction and potentially corrupt financially significant data.

Changes should be tested in a test environment prior to being released into production. If a change is not tested, an explanation as to why testing is not necessary should be present.


19 For 13 of 30 changes, approval by the designated approval authority was not evidenced prior to the implementation into the production environment as required by DOT policy. The applications included in the 13 exceptions were:

• TIPS • FAMIS • BASIS2 • Health & Welfare


Prior to being released into the production environment, changes should be approved by the designated approval authority. The approver should not be the person who requested the change or the person who developed the change unless an explanation is noted in the change management ticket.


20 Program changes related to the Health and Welfare application are not logged in the change management system as required by DOT policy.

Individuals cannot be held accountable and audit trails of program changes cannot be produced if the tracking and logging of program changes is not performed.

All changes that are implemented into production should have a corresponding change management ticket. The change management process should be uniform across all applications.




Page 14 of 21

Computer Operations


21 Formally documented policies and procedures over disaster recovery do not appear to exist. Technical disaster recovery test scripts for mainframe applications were provided, however, these scripts have not been incorporated into formal policy and procedures.

Without formalized policies and procedures in place, management runs the risk of users not following a standard process, resulting in unauthorized activities and transactions as well as ineffective use of resources.

Policies and procedures typically provide clear accountability of roles and responsibilities between staff and management.

We recommend that disaster recovery policy and procedures be documented, approved, and communicated.


22 Disaster Recovery tests are not performed for Windows based applications.

Failure to locate items which are off-site may result in the inability to recover the system during a disabling event.

We recommend Disaster Recovery tests be performed periodically (e.g. at least once a year) to validate that all systems can be restored in the event of a disaster.


23 There is no policy and procedure over backup tape rotation. While tapes appear to be rotated off-site, logging of the off-site rotation is not evidenced to ensure the timely and accurate retrieval of off-site tapes.

Failure to locate items which are off-site may result in the inability to recover the system in the event of a disaster.

When backup tapes are rotated off-site, a log should be created to document the location of each of the backup tapes.

A log has been created to account for this condition. This process will be monitored ongoing for compliance.



Page 15 of 21

Opportunity for Process Improvement Note: The following observation does not represent an issue that would prevent the achievement of the objectives; however, it is provided to assist management in further strengthening their IT internal control environment over financial applications.

No Observation/Condition Risk / Implication Recommendation

1 An annual user recertification is not completed or documented for each in scope application.

Users that are terminated, transferred or promoted may maintain a level of access which is not commensurate with their job responsibilities.

We recommend that an annual user review be completed for each system where management, role owners, and data owners review active users and user rights to ensure that each user’s access is limited to what is required for their job function.



Page 16 of 21

Appendix A: Supplemental Background Information

Overview

The DOT is one of at least 26 different departments within the City of Philadelphia responsible for Information Technology as identified in our interviews with management. The DOT is responsible for the City’s network infrastructure and supports over 100 of the City’s estimated 400 production applications. Of these applications, the following systems were identified as the City’s key financial systems:

• Financial Accounting Management Information Systems (“FAMIS”) • FAMIS subsystems (including ADPICS, the City’s purchasing system) • Payroll • Pension Payroll • Taxpayer Inquiry & Payment System (“TIPS”) – Revenue Accounting • TIPS – Revenue Settlement / Cashiering • Health & Welfare • BASIS2 - the City’s water billing system

All of these systems reside on the mainframe except for BASIS2 which is installed in a Microsoft Windows environment.

Organization Environment Controls (IT Governance)

The Division of Technology was originally established as the Mayor’s Office of Information Systems (“MOIS”) by executive order in September 1993. This order charged MOIS to work in partnership with the City’s agencies to implement information systems that support and enhance the City’s government operations. As a result of this order, MOIS served as a coordinator of technology activities by providing centralized planning, information systems standards, project management, and computer training and support. They also became responsible for the City’s data center, wide area network, Technology Training Center, and the Operations Support Center. This executive order was replaced by Executive Order 08-08 which broadened the responsibilities of the DOT to include “…oversight responsibility for all City telecommunications and information technology functions, projects and personnel.”

During Fiscal Year 2008, the Division of Technology was organized into 4 divisions:

• The Security Division is responsible for protecting the City’s information assets through the development and implementation of information security policies, procedures, and standards. These standards are aligned with the City’s priorities, industry best practices, and government regulations.

• Global Information Services (“GIS”) and Enterprise Technology is responsible for identifying IT solutions that are suitable to be used across the City. In addition, the group develops enterprise policies and procedures for IT governance.

• The Program Management Office (“PMO”) is responsible for the acquisition, coordination, and provision of IT services and resources. In addition, the PMO is responsible for the successful implementation of IT projects. The PMO assists in the development of collaborative and strategic interdepartmental initiatives, policies, and practices for the City’s departments.



Page 17 of 21

• The Operations Division is responsible for the City’s telecommunications infrastructure and the network perimeter, including the primary network architecture with access to the Internet, upkeep and maintenance of all governmental based information technology, as well as web strategy and application development. This group includes helpdesk and system maintenance as well as application development support to the various departments within the City.

While the Division of Technology is responsible for developing enterprise policies and procedures, the implementation of these policies and procedures is the responsibility of the individual City organization providing the IT services. The DOT is responsible for delivering technology services for the applications, operating systems, databases, and infrastructure within their span of control. The responsibility and ownership of processes for many applications is shared between the DOT and the supported business unit. In situations where there is shared ownership of the application, the DOT is typically responsible for the administration of access to the network and application while the business unit is responsible for managing user and transaction capabilities within the application.

System Access and Security Controls (Access to Programs and Data)

Access to Programs and Data represents the processes and controls used to secure the IT environment and its data. This area encompasses the following:

• Security Authorization: The process used to approve access for new users, change access of existing users, and remove user access to system resources.

• Security Administration: The process used to assign user access privileges to the network, operating systems, databases, and applications based on a user’s requested and authorized access.

• Security Configuration: The process to setup, change, and manage security settings that protect the network, operating systems, databases, and applications according to established standards.

• Monitoring of Access: The process to ensure each of the other security processes are operating effectively and consistently according to established policies, procedures, and standards.

According to the MOIS executive order and the subsequent DOT executive order, the DOT is responsible for providing centralized information systems standards. Formal documentation of established policy, procedures, and standards provides a mechanism to ensure that individuals are aware of their responsibilities for securing the City’s computer resources.

The DOT is also responsible for managing system access and security at the network and operating system levels for the financially significant applications included within the scope of this review while the respective end-user communities are responsible for administering the specific data and transactions users can access within the application.

In order to access the financially significant applications hosted on the mainframe or Microsoft Windows environments, an operating system ID is required. The City uses IBM’s Resource Access Control Facility (“RACF”) to manage access to the mainframe and Microsoft’s Active Directory to manage access to the network and Microsoft Windows environments.



Page 18 of 21

Only DOT security administrators can create, change, or disable Active Directory IDs on the servers under the responsibility of the DOT. Security administrators within the DOT and individual end-user communities can create RACF IDs. End-user communities refer to City personnel with IT responsibilities organized outside the DOT and within City operations such as Finance, Revenue, and Water. DOT security administrators have the ability to create a RACF ID and grant access to all mainframe resources, while the security administrators in the end-user communities can only create RACF IDs with access restricted to mainframe resources owned by the respective user community. DOT security administrators require a completed and approved Computer Access Request form before new RACF or Active Directory IDs are created or changes are made to existing IDs. The DOT is also responsible for deleting a user’s RACF and Active Directory ID upon notification of the user’s termination. The DOT is not responsible for deleting or disabling the user’s ID from the application or user IDs created at the application level. The responsibility for deleting and disabling user access within an application is the responsibility of the security administrators within the respective end-user community. While the RACF IDs created by DOT security administrators will permit users to logon to the mainframe, users will not be able to access the applications until security administrators within the respective end-user community grant the necessary access at the application level. The process to grant and control access at the application level varies depending on the application.

FAMIS and FAMIS Subsystems If a user requires access to FAMIS and FAMIS subsystems, a FAMIS security administrator within the Finance Department will add the user’s RACF ID to the FAMIS security table. The FAMIS security administrator will also identify the screens and transactions each user is authorized to access within the FAMIS security tables. FAMIS security administrators also have the ability to create and change RACF IDs. However, FAMIS Security Administrators can only authorize access to FAMIS mainframe system resources to RACF IDs. FAMIS security administrators are responsible for removing a respective user’s RACF ID from the FAMIS security tables upon notification of the user’s termination. Only the RACF ID and a respective entry in the FAMIS security tables for the respective RACF ID are required to access FAMIS and the FAMIS subsystems. FAMIS and the FAMIS subsystems do not require an additional user ID or password.

Other Mainframe Systems The remaining mainframe applications included within the scope of this review use Software AG’s ADABAS database management system. These applications include:

• Payroll • Pension Payroll • TIPS – Revenue Accounting • TIPS – Revenue Settlement / Cashiering • Health & Welfare

In addition to a RACF ID, these systems require a second ID and password defined within ADABAS. DOT database administrators are responsible for creating the ADABAS user ID. A user may have one or more ADABAS IDs based on the number of ADABAS applications the user is required to access. Once the ADABAS user ID is created, the respective end-user application security administrator will grant the appropriate access the user requires within the respective ADABAS application. Each ADABAS application has a separate group of security administrators.



Page 19 of 21

A user of an ADABAS system must first successfully log onto the mainframe using a RACF ID and password. Once a user logs on to the mainframe, the user enters the appropriate ADABAS application path. After the ADABAS application path is entered, the user will be prompted by the system for their ADABAS user ID and password.

BASIS2 Water Billing System The process for granting and maintaining access to the Water Billing System is similar to the process described for “Other Mainframe Systems.” A DOT Oracle database administrator is responsible for creating user IDs to the BASIS2 system. In addition, DOT Oracle database administrators will be responsible for administering application level access to BASIS2. Since BASIS2 is a relatively new system, the responsibility for administering application level access is shared between the DOT and the consultants who were responsible for implementing the system.

The City has established informal minimum password configuration standards for password length and account lockout. These informal standards apply to the network, operating system, database, and application levels.

To protect the City’s data at the network perimeter, the DOT maintains control of traffic coming in and out of the network through the use of managed firewalls. The DOT also performs vulnerability assessments to identify potential network exposures. The DOT performed a vulnerability assessment within the last year to identify further improvement opportunities for the portion of the City’s network under the DOT’s control. An anti-virus solution is also installed to prevent, detect, and remove computer viruses within the City’s network environment. Finally, remote access to the City’s network is controlled through the use of a VPN which encrypts all incoming and outgoing traffic.

Application Development and System Software Controls (Change Management)

The process to manage changes to the production environment is guided by a Change Management Process document published by the DOT. This process, along with a Lotus Notes Change Management System, helps to ensure that changes are authorized and tested prior to their implementation. The DOT is responsible for the development and production support functions for systems included within the scope of this review. DOT personnel are also responsible for promoting changes to the production environment. All changes to these systems are required to follow this process.

End-users or DOT personnel can initiate a change by either placing a call to or sending an e-mail to the DOT help desk. The help desk creates a ticket in the problem management system (c-Support) to document the request. Once the support ticket is created, changes are reviewed and approved by the immediate manager of the individual who initiated the request. If the individuals initiating or approving the change are not members of the end-user community who own the request, appropriate review, concurrence, and approval must be secured from both end-user communities. Once approved, the ticket will be forwarded to the Change Coordinator for submission into the Lotus Notes Change Management System.

The Change Coordinator utilizes the change control log from the Lotus Notes Change Management System to communicate open issues to DOT personnel. Change management meetings are held at least weekly to address any pending changes to the DOT environment which appear on the change control log.



Page 20 of 21

After the change has been completed in the development environment, it is submitted for testing. DOT personnel responsible for coding the change will perform unit testing to ensure any coding errors are corrected prior to user acceptance testing. All changes are required to have user acceptance testing. The user who initiated the change is responsible for performing user acceptance testing to validate that the change is operating as requested. All changes are tested in a test environment which mirrors the production environment prior to the implementation of the change into production. The DOT is responsible for maintaining appropriate documentation to evidence the testing performed and the results of the testing.

Once the changes have been tested, the Change Coordinator must provide a final approval before the change is implemented into production. After the final approval, the Change Coordinator will notify the application database administrator team that the change is ready to be deployed into the production environment. The application database administrator team notifies the Change Coordinator when the change is implemented into the production environment. The application database administrator will also notify the Change Coordinator of any problems encountered as a result of the deployment. If there are no problems encountered, the Change Coordinator will close the change request in the Lotus Notes Change Management System.

Emergency/rush changes can occur when a production system, device, application, or process is failing or has failed and must be corrected immediately. These situations are managed outside of the change management process. For emergency changes, it may not be possible to submit a written request for approval before implementation. However, a formal request must be submitted after the change is implemented and is subject to the same approvals as those changes implemented using the Change Management Process.

Processing, Disaster Planning, and Contingency Controls (Computer Operations)

Computer Operations comprises controls over the continuous operation of the IT infrastructure as it relates to the batch processing, job scheduling, error handling, data backup and restoration, and disaster recovery of systems supported by the DOT.

Physical Security over the Data Center The hardware for many of these systems is stored in the City’s Data Center which is secured via key card entry and restricted to DOT operations personnel. All financially significant systems in the scope of this review are housed in the City’s Data Center. Data Center visitors are required to present a form of identification and sign-in before entering, and are required to sign-out upon leaving the data center. Security cameras are installed at all points of entry to the Data Center and activity is recorded 24 hours per day, 7 days per week.

Environmental Controls over the Data Center Environmental controls are in place to protect the infrastructure and data in the data center. The DOT maintains heating, ventilation, and cooling (“HVAC”) units that are dedicated to the Data Center. The DOT maintains a maintenance contract for the Data Center’s HVAC with a reputable vendor. A Uninterruptible Power Supply (“UPS”) system provides temporary power to the Data Center in the event of a power outage. In the event of a fire in the data center, a sapphire-based fire suppression system will release compressed gas to extinguish the fire. The fire suppression system is certified twice per year by a reputable vendor.



Page 21 of 21

Backup Procedures

Mainframe Applications

The DOT performs incremental backups of all mainframe application production files on a daily basis Monday through Saturday. The DOT Operations team performs a full backup for all mainframe production files on a weekly basis. Three generations of daily and weekly backups are maintained. In addition, monthly backups are performed for each of the weekly backups which were performed during the course of the month. The monthly backups for all mainframe production files are maintained for one year.

The backup tapes which are stored onsite are located in a robotic silo within the data center. Mainframe backups are rotated off-site weekly, and a member of the Operations team is assigned to transfer the tapes to the location via car. The tapes are stored in a fireproof case when the tapes are transferred to the off-site locations.

BASIS2

The DOT utilizes a backup and recovery manager supplied for Oracle databases by the Oracle Corporation called RMAN. RMAN is configured to create change logs on a server located in another City of Philadelphia building. The log records all changes made to the database. If there is an error in a data file, the logs can be used to restore the files to the point prior to the failure. The RMAN logs are then backed up to tape library on a daily basis. Full backup of all of the data files is performed and copied to the tape library on a weekly basis. Due to the real-time backup process for BASIS2, BASIS2 can be restored from the change logs stored on the server located in the other building. Because of DOT’s ability to restore BASIS2 from the change logs, the tapes are not rotated off-site.

Disaster Planning and Contingency The DOT has contracted a third party service provider for recovery of the mainframe in the event of an emergency. The DOT’s mainframe backup tapes would be taken to the third party’s location in the event of an emergency and restored to the mainframe at the third party’s facility. Disaster recovery tests are performed by DOT’s technical group at the off-site location four times a year. The DOT maintains documented step-by-step procedures of the disaster recovery plan. Periodically, DOT personnel who are not members of the technical group will participate in the disaster recovery test to ensure that the testing instructions are accurate, can be followed, and satisfy the needs of the business in the event of a disaster.

Documents

General IT Cotnrols Review of the Division of · PDF fileCity of Philadelphia Division of Technology 1234 Market Street, Suite 1850 ... Susan LaCour Technology Manager BASIS2 Security