Embed Size (px)
Citation preview
TAFE SA
Starting from scratch
An architecture and lessons learnt
Hamish van den Ijssel
Melissa Hazelwood
APSUG - September 2010
Overview
• TAFE SA project overview (5 minutes)
• Architecture (20 minutes)
• Lessons Learnt (10 minutes)
• Questions (10 minutes)
TAFE SA
• 85,000 students – 25,000 FTE
• 53 campuses, APY Lands, overseas
partners
• VET and tertiary students
• Relationships - TAFE SA, DFEEST and
South Australian Government, Federal
Government
ICT Overview
• Decentralised -> Centralised
• Outsourced components
• No SungardHE exposure
• Very limited Oracle and Unix exposure
• Existing application suite circa mid
1990’s
Project Overview
• December 2009 – Test hardware delivered
• February 2010 – First logon to Banner test environment
• January 2011 - Disaster Recovery Testing
• February and March 2011 - End to End testing
• April 2011 - CORE release - Registration– INB, SSB, ODS
– Cognos, eVisions, ePrint
– Interfaces and reporting
• June 2011 – Admissions– Interfaces and reporting
• August 2011 – March 2012– CORE +– Workflow, BDMS, Luminis
– Reporting and interfaces
– IAM
• July 2012 onwards – CORE++– Post implementation projects
Architecture• Primary Site – State government data centre
• Secondary Site – Tea Tree Gully Campus data centre
Primary Site Secondary Site
Database Servers
Primary SitePrimary SitePrimary Site
EVA8000
Primary SitePrimary SiteSecondary Site
EVA8000
• HP BL860’s – Oracle 11GR2 RAC – HP-UX 11.31
• HP EVA8000 and EVA 8100
Application Servers
Primary SitePrimary SitePrimary Site
EVA8000
Primary SitePrimary SiteSecondary Site
EVA8000
• HP BL460’s VMware vSphere cluster
• RHEL 5.0, Windows 2003
• Oracle Application Server 11G
Virtual Hosts• External facing – SSB + Luminis*
• Internal facing – INB, ePrint, BDMS, Cognos, WF, eVisions, Grid Control
Primary SitePrimary SiteSecondary Site
EVA8000
SSB1 SSB2
SSB3 SSB4
External DR
INB1 INB2
ePrint /
eVisions
BDMS
GRID
Control
Internal DR & Dev/Test
Cognos
WF eVisions
Primary SitePrimary SitePrimary Site
EVA8000
External
SSB4 SSB6SSB5
SSB2 SSB3SSB1 INB1 INB2
ePrint /
eVisions
BDMS
GRID
Control
Internal
Cognos WF
Database Utilities
• ADDM
– Automatic Diagnostic Monitor
• AWR
– Automatic Workload Repository
• Oracle Grid Control
Utilities
• Backup – IBM Tivoli
• SAN replication software
• VMware Site Recovery Manager
• Oracle Load Testing Suite
Load Testing
• Goals
– Validate system architecture
– Identify bottlenecks
– Performance tuning – Network, Servers and
software
• Issues
– 53 campuses; Performance at remote campuses
– Complexity of the environment
Load Testing
• Oracle Application Testing Suite
• Integration with Oracle Grid Control– Oracle RDBMS
– Oracle IAS
• MIB files– Servers
– Operating Systems
– Database Servers
– Application Servers
– Routers
– F5’s
– PC’s
– Firewalls
Load Testing
• Phase 1 – June 2010– Development hardware
– Validate system architecture
• Phase 2 – October 2010– Identify issues with regional networks
• Phase 3- November 2010 – Primary site hardware with 80% configured
– Performing tuning
• Phase 4 – February 2011– Primary site hardware with 100% configured
– Performance tuning
• Phase 5 – August 2011– Secondary site hardware with Banner 8.X
– Performance tuning
Load Testing Example
• 1600 virtual users
• 3 separate test scripts
– Apply for admission
– Check registration
– Course Search
• All test scripts utilise variables via datafiles, i.e. Username, password, course
All transactions versus time
Number of transactions over time
Errors over time
Errors
• Excessive time for transaction to complete
• Insufficient time between each step of the test script
• Web server busy; unable to serve page
• Excessive time for post / send
• Database busy / slow
• Network traffic / issues
• OATS Test step expected and actual result different
Performance over time
Total time for transaction to process per millisecond over the life of the test
Agents session counters
Active users over time
StatisticsName Min Max Avg
Active Virtual Users 0 1600 1159.82
Virtual Users with Errors 0 823 72.136
Transactions Per Second 0.189 22.796 8.47
Pages Per Second 0.533 110.133 43.39
Hits Per Second 0.533 126.733 49.106
Kilobytes Per Second 9.166 1175.37 412.791
Totals
Transactions 9021
Transactions with Errors 4256
Pages 46210
Hits 52298
Kilobytes 439622
Name Min Max Avg
ApplyForAdminLTST_SSB 5.982 362.784 113.482
CheckRegoLTST_SSB 6.074 360.808 89.663
CourseSearchLTST_SSB 7.377 491.806 150.258
Network
Disaster Recovery
• Network devices– F5
– Fibre
– IPS
• Database– SAN
– Servers
– Data guard
– SAN replication
• Application Servers– Virtual images
– Hosts
– Site Recovery Manager
Interfaces
CORE
– 15 Applications
• 8 Localisations / 7 Custom
• 5 External / 10 Internal
• Primarily Process Rules Engine
– CORE+
• 3 Applications
– 3 (replacement) SPML interfaces
– 2 likely Process Rule Engine
Training
– 59 days / 152 staffing days of training• Sungard
– Banner General and Security ( 3 days – 4 staff)
– CTOOL (2 days – 3 staff)
– Intro to Banner Administration ( 3 days – 2 staff)
– Oracle DBA Administration (3 days – 2 staff)
– Oracle, PL/SQL, and Database Objects (3 days – 4 staff)
– Student Module Technical (3 days – 4 staff)
– Advance DBA Administration (3 days – 2 staff)
– Oracle backup and recovery (3 days – 2 staff)
– Student Data Mapping for Data Migration (GP-Synchronisation (3 days – 1 staff)
– Oracle Application Server (3 days – 4 staff)
– Super Table (2 days – 2 staff)
– Process Rules Engine (2 days – 3 staff)
Training
• Sungard continued…– Value Based Security (2 days – 2 staff)
– Web Tailor (2 days – 3 staff)
– Luminis I & II (4 days – 2 staff)
– RAC / ASM (3 days – 2 staff)
– ODS (3 days – 3 staff)
– OATS (1 day – 3 staff)
– Luminis I & II (4 days – 2 staff)
• Reporting– eVisions (1/2 days – 2 staff)
– ePrint (1/2 days – 2 staff)
– COGNOS (3 days – 3 staff)
– Regulatory Reporting (3 days – 2 staff)
• Future– Grails
– Handover
Training
– Information overload
– Project schedule – training versus workload
– Documentation
– Room bookings – Functional and training
Lessons
What went wrong and what went well
• Licensing
• Decentralised and outsource services
• Security Policies
• Upgrades
• Policies and procedures
• Key Relationships
• Human Resources
• Load Testing
• Vendor Delivery
Licensing
• Include with project tender
• Know vendors end of sales quarter
• Holistically evaluate pricing models and
support options
• TCO
Decentralised and outsourced
• Decentralised business and ICT
– 4 ICT units
– 53 campuses
• Outsourced ICT
– System Administration
– Networking
• Architecture decisions
Resourcing
• Recruitment
– Public service merit selection
• Project staff
– Sourced from the business
– Sourced from industry
– Sungard consultants
– Contractors
• Project as a separate entity from ICT
Security Policies
• South Australian Government ISMF
– Out of date policies
– Incompatible with business practices
• Outsourced security advice
Upgrades
• Desktops
– Windows / Web browsers
• Servers
• Banner
• Oracle
• Functional testing
Policies and procedures
• Policies and procedures
– When, where and how?
• Documentation
– When, where, how
• Release / patch / change management
Original SAR to GO-LIVE
• System Architecture design in October 2009
• Technology incompatibilities
– New technologies
– New releases of Sungard products
– Changing WOG policies
– You don’t know what you don’t know
– System administrator design to holistic design
Relationships
• Functional
• Training
• ICT
• Sungard
• Vendors
Staffing
• Technical Team staff (7.0 FTE)– Internal (2.0)
• Two analyst programmers
– External (3.0)• One Analyst Programmer
• One DBA
• One DBA / Analyst Programmer
– Sungard - long term (2.0)• Senior DBA
• Senior Technical Consultant
– Sungard – short term• Short term fly-in
– Non project ICT staff• Infrastructure
– Vendors• System Administration
• Networking
• Public service merit selection
• Project as a separate entity from ICT
Load Testing
– Objective and informative data
– Easy to use product
– Excellent support from the vendor
– Excellent integration with Sungard products
– Extensive interest from all areas of ICT
Questions
…..
