WPID F1.1-M Lessons Learned Jun09

WPID F1.1-M Lessons Learned / Retrospective

1. Projects InformationProject ID: PG193 (121674) Project Name: Sales FMO AVPN Ph1Project Install date: 06/14/2009 Project WBS:Organization: Program: FMOLife Cycle Phase:Attendees or Contributors Name Area of RepresentationBob Mayo ETE PM

Project ID: PG195 (121655) Project Name: Sales FMO New FunctionalityProject Install date: 06/14/2009 Project WBS:Organization: Program: FMOLife Cycle Phase:Attendees or Contributors Name Area of RepresentationSanju Thomas ETE PM

Project ID: PG196 (121656) Project Name: Sales FMO MIS-AKPRVIHI Ethernet TProject Install date: 06/14/2009 Project WBS:Organization: Program: FMOLife Cycle Phase:Attendees or Contributors Name Area of RepresentationAram Kratlian ETE PM

Project ID: PG198 (121680) Project Name: Sales FMO MIS MACD

PPA-IUP-WBTM-25 Proprietary Information

Not for use or disclosure outside AT&T family of companies except under written agreement.

Page 1 of 26


Project Install date: 06/14/2009 Project WBS:Organization: Program: FMOLife Cycle Phase:Attendees or Contributors Name Area of RepresentationSreenatha Gaddalay ETE PM

Project ID: PG199 (121673) Project Name: Sales FMO Fallout DefectsProject Install date: 06/14/2009 Project WBS:Organization: Program: FMOLife Cycle Phase:Attendees or Contributors Name Area of RepresentationPramod Duttargi ETE PM

Project ID: PD511 (117899) Project Name: Global Managed Services Part1Project Install date: 06/14/2009 Project WBS:Organization: Program: FMOLife Cycle Phase:Attendees or Contributors Name Area of RepresentationMyrna Cadelina ETE PM

Project ID: 117899f Project Name: DUNS Matching for Global Managed Services Part 1Project Install date: 06/14/2009 Project WBS:Organization: Program: FMOLife Cycle Phase:



Page 2 of 26


Attendees or Contributors Name Area of RepresentationSavita Rajeevlochan ETE PM

2. Project Evaluation and Accomplishments PG193:

PG193 did not complete all phases of testing prior to release. AT&T gave the approval to proceed to delivery and was delivered to production successfully on the scheduled date. UCT was a ‘Go’ with no outstanding issues on UCT day. Testing was completed through ABR processing during the release and then through SOR on the Monday after release. After completion, the test case was cancelled

PG195:This project is important to ATT business to make sure that users are led down an intuitive and easy path to generate quotes, contracts and orders

Customers are able to electronically sign contract documents Authorized AT&T resources are able to electronically countersign contract documents Sales not required to re-key data that has already been entered in other systems Users are able to generate standard reports and create ad hoc reports/ad hoc queries without engaging IT / Labs Administrators are able to maintain and configure templates without engaging IT/Labs Minimal training required for sales and support

PG196:This project delivered two main deliverables: 1) Support for MIS in Alaska, Hawaii, Virgin Islands and Hawaii, including MACD, across the FMO platform. 2) Domestic Ethernet for MIS. It was approved for installation by AT&T and IBM release management and was successfully implemented on 6/14. However, JST and UAT did not complete but will continue testing through the end of June. This is with the full support and approval of AT&T FMO.

PG198:



Page 3 of 26


PG199:

PD511:

117899f:

3. Estimating and Planning Resources Evaluation

Required for ALL Projects:

PG193 :

Planned Delivery Date

Actual Delivery Date

Planned Hours or FTP

Actual Hours or FTP (through 6/19/09)

Project Level 06/14/2009 06/14/2009 131.87 ftps 139.41 ftps



Page 4 of 26


Reason for Variance outside +/- 20 for Large Projects (Greater than 3 FTPs) and +/-30% for Small Projects (Less than 3 FTPs) of organization objective."

Variance is within limits. Less than the +/- 20% variance

PG195:




Actual Hours or FTP




PG196 :







Page 5 of 26





PG198 :








PG199 :



Page 6 of 26









PD511 :










Page 7 of 26


117899f :








4. Quality Results Evaluation

PG193 :

CR SummaryCR011 AVPN Integrated Services- ETE-PE140, PE141, PE139, and PF035CR026 AVPN UpdatesCR056 EFMS-PM UpdatesCR087 EFMS-PM Web service UpdateCR095 AVPN Service AvailabilityCR129 setProjectPackage to Support RDSCR266 Remove Vendor Sold Notice Requirements for JRD and swBRD for PG193. This is a documentation only change to align the JRD and swBRD with the agreements made and



Page 8 of 26


implementation moving forwardCR271 Make changes to the generateActivityGroupReport() web services on the request and response blocks. eCRM has a need to query based on MDS ID to return all of child Activity Group Reports for the MDS ID. CR308 Update is required to the AVPN Pricing Schedule.CR312 Missing PG193 AVPN RDS fields that need to be captured by ADOPT and GCP.CR313 Collecting an AVPN discount for the Sub-Categories levels.CR379 ECRM HLD UpdatesCR380 ICT & CMT Data WSDL UpdateCR394 Support Requests-Remove BCS Specifics from ADOPTCR404 eCRM code change for CustomerID”.CR428 Make BillingAddress and Contactblocks optional for setBillAccountPackage and getBillAccountPackage. Fixing R209 JST QC-11828 Ticket

SQA Summary An SQA audit was conducted in the R&F & Design phases – Passed.

Release Management Project was released through Release Management and was successfully deployed on 6/14/09.

Test Summary All features were ST to completion. JST test was conducted and to be completed post release. UAT did not complete prior to the June release and then continued post release. ATT gave the go to continue to UCT in the July Release. CRs313 and CR381 were deferred to project 121680a for the July release

ITO - Information Technology Operations (includes Managed Operations / PSO as applicable)E-Governance as applicable

PG195

CR Summary CR033 Custom Pricing CR



Page 9 of 26


•DBOR in custom pricing process for Solution XML should return the Solution URL to eCRM. DBOR should not be processing spreadsheet

CR126 Custom Pricing SD Update•SFD Document update to reflect the following changes: the preconditions to "ReadyForContractGeneration" needs to be stated and clarified, the hierarchy of the price types (i.e. budgetary, icb, firm) needs to be stated as well, for the Resubmit - we'll be reusing the same pricing scenario as in the original request

CR223 A new MIS Pricing Schedule (PS) to support SEU and SEUL Single Site promotions with MLPPP is being provided for the R209 release (unsure which project this will fall into). This template simply adds verbiage to support a single site using the standard MIS PS.

CR227 Whenever a MSA/UA is generated, the letters “UA” need to be appended to the system-generated AT&T MA Reference No.

CR257 For code impact – there is a need, due to other PG195 requirements, to update eCRM WSDLs.This may impact:ABN on the Web (BeSales),eGBS,eSIGN,IMS2,SalesOne, IGLOO.

CR263 Use BGW as a gateway between GCP and CADM for CADM real-time data. GCP will need a web service interface with BGW to retrieve the CADM data required for the getContractInventoryDetails, getContractInventorySummary and getBillAccountList web services.

CR291 ADOPT has to implement Express Orders for the R109 February 2009 Release. Cover additional scenarios when COS is YES for Basic offers and No for Plus (managed offers) of express hadling For TOI Derivation rules, please refer to the “TOI Derivation - OI_AD_AJ_01072009.xls” Excel file.

CR292 In eSign, instead of giving the user (AE) a backend process to generate the custom language support request, client is requesting to be able to launch from eSign to eCRM to create the custom language support request that is currently in use today.

CR293 CTS Performance Testing requested to measure the response time of key transactions in eCON, review with the users and tune the application as needed prior to the R209 release in June.

CR299 IMS2 Credit ElementOPS/ADOPT will read the pricing results (discounts, rates, credits, etc.) of the custom pricing activities and present those results to sales

CR315 The proposal for change provides:a) eCON to send eSign the selected ODM information for a contract negotiation at the contract group level. b) eSign will display this information on the Deliver Contracts screen.

CR316 Restart/Error Recovery was not in the original scope. i. Application (eCRM, ADOPT, eCON, eSign) will trap the error at the point of failure in the Contract Negotiation flows (Move, Remove, Update, Custom Language) and report to C-BUS.

CR321 In order to maintain availability for FMO, a snapshot copy of CADM will be created. Ordering activity will continue using the snapshot, while billing initiation activity takes



Page 10 of 26


place on the original copy of the dataCR322 COBRA Biller Code Block

Ensure MIS/BVOIP account setups are not done using COBRA biller triplets. CR326 Needed to support SE&A Activities for the design checkpoint/CDR (produces the data

flow diagrams (DFDs), test plan reviews, test sets reviews, Root Cause Analysis) during JST and UAT.

CR338 Create a new tab on the eSign Customer List call “Archive”. The AE would be given control on all other tabs where eSign requests exist to select one or more eSign requests and move them to an Archive Tab and the ability to move the request(s) back to the appropriate status.

CR346 BGW needs to provide the PHI instead of Component ID when providing the IDs for plans and charges. This includes Pricing Plan, Contract, Discount, Rate Plan and Inventory items as specified on the queries from project PG195 and PG198. BGW will need to derive these IDs for plans using the SA FBO table and existing logic used for other projects.

CR347 ADOPT has identified the following 4 Documentation Only updates that need to be made to the ADOPT HLD & AID.1. HLD -label name changes 2. AID - ADOPT Contract Negotiation to remove the error-codes3. HLD-3.1.2: Change the HLD Requirement Description.4. ADOPT-CBUS AID for PricingScenarioStatus Event & ToDoActivity Event.

CR359 GCP PricingScenario IAD update for the change from <AssociatedCustomerLocationXref> to <AssociatedCustomerSiteXref>

CR376 SRD changes for PG195 ECRM SRD to -• SetOpty Clarifications• Todo Delete Button Added• Removed Assigned to in Emails for ToDo• ICB SR - Rules on how to refresh the records• Review Board/ Capacity Check Clarifications on Approval Effective Date

CR402 eCRM changes from PG195 DemoCR410 This PG195 CR is for GCP to cleanup DPPCO records to prevent duplicates when

ADOPT writes same product component data into GCP at different timesCR413 This CR is for the following changes to be made to eCRM and eCON. (These changes

were made during the design phase. This is a documentation CR to update the swBRD to match the design & coding)

CR441 eSign Full Browser Window

SQA Summary An SQA audit was conducted in the R&F & Design phases – Passed with no Non Conformances or Non Compliances.



Page 11 of 26



Test Summary All features were STed and JSTed to completion. UAT did not complete prior to the June release and then continued post release.


PG196

CR Summary CR372 Update HLD requirements to reflect changes suggested by development team during coding.CR373 Add DSM-FEDI as test support only.CR258 Add GIOM and EFMS to project to reflect feature changes to HI, AK, PR, VI

SQA Summary An SQA audit was conducted in the R&F & Design phases – Passed.


Test Summary All features were ST except for two test cases involving GIOM and eCRM. The root cause of not completing was in inability to secure test data from ADOPT. Testing will continue post-implementation with an expected close date of 6/30.. JST test was conducted and completed except for two test cases. One involves defect 13398 which will require the origination of a CR (478) to be addressed in Project 121680a. The other involves testing with HSAT (ATT application) which is planned to test through the end of June. If it does not complete, then a CR will be initiated to also continue processing in 121680a.



Page 12 of 26


UAT did not complete prior to the June release and then continued post release. ATT gave the go to continue to UCT in the July Release.


PG198

CR Summary CR018 PB732 Impacts

•MIS Replacements and A&BR/Contract Automation - it was uncovered late during Phase 2 timeframe that there is a project targeted for June (PB732) which automates some additional contracting areas for MIS.

CR047 A&BR Term Length on Renewals•This CR is being defined to address an identified issue with the current A&BR implementation of MIS renewal contracts. Currently, in GCSM, a contract to simply extend the MIS contract for the same term length (eg. 1 year to new 1 year, 2 year to new 2 year, etc.) without any other contract changes is handled as a "renewal/addendum".

CR105 ADOPT-eCRM Updates•During the Design phase discrepancies were found in the ADOPT and eCRM requirements that require swBRD updates. These changes are documentation-only updates to maintain consistency between the swBRD and SRDs.

CR193 GIOM MDS ID•GIOM needs to store the MDS ID (ForeginSorId) retrieved on getActivityGroupPackage for the BVOIP order, display MDS ID on Order Details Windows, link shopping cart orders to MDS ID, and allow searches based on MDS ID.

CR242 CADM Impact for MIS MACDCADM was originally listed as test-only impact. CADM is a critical path for MIS MACD in retrieving the contract inventory details and is code impacted.

CR248 Data length from upstream application (eCRM) is not matching GIOM data lengths. GIOM's data lengths match SOR downstream application data lengths.

CR263 Use BGW as a gateway between GCP and CADM for CADM real-time data. GCP will



Page 13 of 26


need a web service I nterface with BGW to retrieve the CADM data required for the getContractInventoryDetails, getContractInventorySummary and getBillAccountList web services.

CR270 1) In order for ADOPT to receive a set of MIS inventory locations based on a contract/service triplet first, GCP to create a new synchronous (real-time) MIS getServiceInventorySummary web services.2) INSTAR was originally listed as test-only impact but INSTAR data analysis is a critical path for MIS MACD in retrieving the MIS inventory details.

CR279 Remove following requirement from JRD and swBRD from the scope of project PG198:B-4.1.2.27: Lifecycle Ordering for MIS/PNT IPTO Contracts.

CR319 Current design allows users to select multiple contracts in eCRM ALP screens. Since this data is transferred over to ADOPT when the user is launched into ADOPT, eCRM will limit this transaction to 10 contracts for replacement purposes

CR351 1) rerun the PF703 Clarify and USRP scripts to populate the EFMS generated orders inputted after the initial PF703 was implemented, and2) allow Clarify to issue billing disconnects on EFMS generated order

CR392 MACD Functionality - requested scope changes to PG198 MACD functionality. (Scope reduction)

SQA Summary An SQA audit was conducted in the R&F & Design phases – Passed with no Non Conformances or Non Compliances.


Test Summary As a ADOPT was upstream system and most of MACD functionality from ADOPT was not ready until code deploy, so ST was completed on simulated data. Extended UAT and PDUAT was carried in 121680a




Page 14 of 26


PG199

CR Summary CR246 GCP code impact from eCRM work for TR-2008-06-1006 and BR-2008-06-1011CR264b Fallout - Defect # 8127

As a result of accommodating navigation and approval options, ADOPT and eCRM need to share the same SR status, as well as option approval and expiration detail.

CR277 Impacted organizations, systems, etc.: MIM Code to generate user files from CENET Data feeds to eCON and ADOPT

CR296 This CR is required to reduce cycle time delays & eliminate SOR defects & manual handoffs due to missing site level information.

CR340 BDM Discount RestoreRe-instating the BDM envelope with up to 100% discount authority due to a requirements misinterpretation.

CR348b Defects 9781, 9782_BO Report FixesCR349 Defect 8950 and 9073 eCRM dashboard changesCR350 DEFECT 10046 – 'PF035-TC29 Customer Name Field delete, Account ID field moveCR353 Defect ID : 11147 i. PE141 - eCRM BO Report Common tab - missing remarks field.

There is no place to enter Project Level notes in ADOPT. There is no Project level notes entered in eCRM.

CR354 AOTS 113191696: Contract Type on "pseudo" contracts not populating on BO report. eCon to pass the contract type to GCP for IPT contracts, (including “pseudo” contracts) . The data element is already in the interfaces.

CR363 DEFECT 9465 – TC25 - Stare and Compare flag is not set when ADOPT SDA discounting is applied in ADOPT

CR384 Defects 113326735, 10958, 11198, 7342, 8556, 10047 and 10574CR388 Required for ASAP DPP to maintain parity with PMO.

ASAP DPP needs to increase the number of sites permitted in IPTO MIS, IPTO MIS with Managed Router, IPTO PNT, and IPTO IPFR deal types.

CR389 Sales Stage and Num of Physical Locs1. Capture the sales stage of Opportunity at the time of SOR creation.2. For New ABN, capture ‘# of Physical Locations’ at service type level also.

CR391 GCP OLAP impact from Emptoris 7.5 Update that will impact code for PF701 (March 09 release). CR393 MARO Fixes.

1. Update ADOPT Order Solution screens for MARO to emulate current Design screens and functionality. 2. Update ADOPT MARO templates to correct configuration rules. QC Defect 8688.

CR400 This CR will handle the issue where the BO Reports are not being locked in FMO as they are in the current PMO implementation



Page 15 of 26


SQA Summary

Release Management

Test Summary


PD511

CR Summary Cr 63 UAT Defect 14324CR 56 CR 56 - ASOC & GCP changes; reverses ASOC WSDL code for base projectCR 58 CR 58 - IES Descope

SQA Summary

Release Management

Test Summary



Page 16 of 26



117899f

CR Summary

SQA Summary

Release Management

Test Summary




Page 17 of 26




Page 18 of 26


5. Lessons Learned by Phases

Phase 1

What Worked

Issue Reference Root Cause Resolution Description Key Learnings Candidate for LL-DB

What Didn’t Work


PG193/PG196/PG195/PG198: Phase 1 rushed

swBRD was delayed Delay in swBRD caused a ripple effect through out the entire schedule causing much churn to the project.

We were in a position where there was little we could do but try to get resolutions



Page 19 of 26


Phase 2

What Worked


PG193/PG196/PG195/PG198 : P2 Partnering

Good partnership Good working relationship with ATT PM

Continue to partner with ATT PM and work together

PG193/PG196/PG195/PG198 : P2 Communications

Good communication Numerous meeting were held to go over issues

Issue were resolved in meeting much more quickly than numerous emails

ALL: P2 Status Joint IBM / ATT meetings Weeking meetings held jointly with ATT to discuss all R209 Project status

Continue to work closely with ATT at a Program level

What Didn’t Work



Page 20 of 26



PG193/PG196/PG198 : P2 – Constant Churn

Late requirements Delay in swBRD caused a ripple effect through out the entire schedule causing much churn to the project.

We were in a position where there was little we could do but try to get resolutions

PG193/PG196/PG198: P2 – Document baselining delays

Difficult time getting ATT Tier1 to baseline HLD documents

Required many follow emails, Q Messages and meetings to get the necessary approvals

Engage AT&T PM to get the AT&T votes

PG196: Project Mgmt

PG196 experienced 3 different AT&T project managers throughout various phases of the project. Each had a different style and expectation and we lost project momentum at time.

Required constant coaching and training and extra work to orient the new ATT project manager which took away time from actually managing the project. Once the third PM was assigned, some stability returned

Ask AT&T to minimize switching off project managers throughout the project lifecycle.



Page 21 of 26


Phase 3

What Worked


PG193/PG195/PG198 : P3 App Support

Need elevated support for this time crunched project

Good support from App teams to meet schedules.

Establish good communication with App teams

PG193/PG196/PG195 : Issue Resolution

Issue or questions that surfaced were aggressively resolved.

Application teams became more efficient at giving weekly status.

Engage the Apps and help them to understand the Customer desires for status

What Didn’t Work


PG193/PG196/PG195/PG198 :

Compressed Schedule Due to the compressed schedule of this project and

We were in a position where there was little we



Page 22 of 26


P3 – Constant Churn

the numerous CRs, dates were often missed and re-negotiated. Testing was extremely compressed with little chance of completing

could do but try to get resolutions

PG193/PG195 : P3 – PRISM CRs/Est

Many CRs causing lots of schedule churn

Getting CRs created in PRISM and getting estimates

PG196: App Support

Most application failed to be responsive in providing audit documentation on a timely basis. Some artifacts, labeled baselined were, in fact, not baselined.

Escalated to 1st and 2nd line managers and discussed at executive meetings.

Ensure teams know what is expected from them and when for audit artifacts. Escalate sooner.

Phase 4

What Worked


PG193/PG196/PG198: P4 ST Completion

ST was managed by Project Test Manager

ST was managed by Project and Program Test Managers

Ensure Project and Program test Managers are engaged early in the project

PG193/PG196/PG195/PG198:

UCT on Release Sunday UCT was managed by ATT and they were quick to get

Continue to work with ATT to provide them the



Page 23 of 26


P4 – UCT the appropriate support team to the call and get issues resolved

contacts for UCT.

What Didn’t Work


PG193/PG196/PG198 : P4 UCT

JST did not complete Due to compressed schedule and ADOPT completion issues, JST was unable to complete testing prior to release

PG193/PG195 : P4 UCT

UCT ran several hours over schedule due to late start with GCP.

Had a communication gap with GCP and their status

GCP needs to be better prepared with their status communication to ensure all projects are notified.

PG193: P4 UCT User ID not properly set up The test case had to be restarted after several hours of processing due to the improper set up of the User’s ID

AT&T UCT testers need to understand the test cases and proactively have all user id properly set up.

PG196: P4 UCT Project test manager was not involved enough across all testing environments. Many testing issues were brought

Set expectations on roles and responsibilities on future projects.

Get clarity on roles at project inception.



Page 24 of 26


to the attention of the ETE PM to address that normally would have been handled by the project test mgr.

Phase 5 - Project Closeout

What Worked


What Didn’t Work

Issue Reference Root Cause Resolution Description Key Learnings Candidate for



Page 25 of 26


LL-DBPG193/PG196/PG198 : Project Close out

Delay in getting TC2 approved through PRISM

ATT has been having trouble getting estimates approved in PRISM

Start the TC2’s in the proper phase

PG195/PG198 Delay in closing out PRISM gates

Apps needs to be more responsive in closing out PRISM gates

Ensure teams know what is expected from them

This project supports a July Point Release for delivery of functionality across a suite of applications used by the AT&T business field sales teams for domestic and most of world (Impacts AT&T Corp). The goal of the Sales FMO (Future Method of Operation) Program is to effect a transformation from our current method of sales operations to a world leading customer and selling experience that provides a significant competitive advantage in the marketplace. The June release cannot support all of the required functionality so change requests have been identified to be moved to July. The applications involved support Sales Opportunities, Design, Price, Propose, Contract, Order and Service Delivery. Functionality being added is to support existing products and enhancements. The SOR experience for Sales should be consistent for all products, intuitive, minimize rekeying, and flow pre-sales information to downstream ordering and provisioning systems. 121680a implements Sales FMO MACD functionality for Add BVOIP, Replace and Reprice plus Remote Worker Enhancements and defect fixes.

08/30/2009

Target Deployment Start: 07/17/2009 Target Deployment End: 08/30/2009



Page 26 of 26

Documents

WPID F1.1-M Lessons Learned Jun09