Data Extracts & ReportingRecent Issues

ERCOT Information TechnologyData Extracts Working Group11/27/07

Agenda

• TML API Outage 9/29/07-10/1/07• Market Notice Review

– M-A103107-06– R-A110907-01– M-A111207-02 – M-C111407-02– M-A112107-03

• Update regarding the search for a new EIS manager

TML API Issue

• Issue:  Market connectivity issues to TML via the API– Issue Dates: 9/29/07 – 10/1/07

– Market Notice: Issue was not experienced market-wide.

– Root Cause: A greater than allowed number of connections to the API caused the Enterprise Application Integration (EAI) layer to refuse some connection requests.

– Resolution: Increasing the number of connection threads in the EAI layer allowed all requests to be handled appropriately. Monitoring: ERCOT has added additional monitoring to capture and correct this prior to becoming a market impacting issue.

10/29/07 - 11/26/07 Extract Issues

• Issue:  ERCOT experienced delays with posting several data extracts– Notice Dates: 10/31/07-11/9/07

– Market Notice: M-A103107-06

– Root Cause: ERCOT batch process delays due to Oracle bug when running database statistics.

– Resolution: Utilize Oracle recommended workaround for running database statistics.

• Further information and timeline on next slide

10/29/07 - 11/26/07 Extract Issues

Market Notice: M-A103107-06 (cont’d)Incident Timeline

• 10/21 – Server consolidation performed during maintenance outage• 10/27 – Database statistics fail – first time running statistics on new platform, causes batch

process to run slowly• 10/28 – Daylight Savings Time adjustment issue with batch scheduling application • 10/29 – Partial rollback of database statistics

– Severity 1 Technical Assistance Request opened with Oracle– Batch process continuing to run slowly

• 10/30 – Events– 0615 – decision made to rerun database statistics, batch held from 0745 until 1030

when statistics estimated to have completed– 1030 – reinstated batch while statistics were still refreshing– 1915 – terminated statistics refresh, attempted statistics with smaller database sample – 1945 – full rollback to old statistics – batch performs at expected levels

» Also began making adjustments to batch schedule in attempt to make up time which caused issue described in Market Notice M-C111407-01

• 11/5 – Estimated catch up date• 11/9 – Actual catch up date• 11/16 – Oracle confirms issue that caused statistics refresh to run for so long was due to a

software bug

10/29/07 - 11/26/07 Extract Issues

• Issue:  Additional ESI ID Extract to post from some Market Participants – Notice Date: 11/9/07– Market Notice: R-A110907-01 Extracts - Retail – Root Cause:

• The 727 delete objects (esiidservicehist_delete, lschannelcutheader_delete, esiidusage_delete) only pull rows that have been actually deleted.  The extract views explicitly only pull rows that have an ov_row_status of 'ID' or 'UD‘.

– Resolution: • Removal of the " and ov_row_status in ('ID' or 'UD') " criterion from the

delete extract views.  To correct the missed rows, a supplemental extract is created.

10/29/07 - 11/26/07 Extract Issues

• Issue:  ERCOT is experiencing delays with posting several extracts – Notice Date: 11/12/07-11/13/07– Market Notice: M-A111207-02 – Root Cause:

• Complications with the batch scheduling software encountered while completing an ERCOT requested rerun of the 11/8 batch (10/31 trade date) resulted in the 11/8 batch running on Sunday 11/10 and completing on 11/11. We moved forward with the 11/9 batch and it completed at approximately 5:00 p.m. on Saturday. When Lodestar batch ran out of order, the controls table was not accurately populated.

– Resolution: • The control table rows were manually entered. A SIR has been

requested to modify the control record creation process.

10/29/07 - 11/26/07 Extract Issues

• Issue:  Data for operating day 10/31 was not included in the Load Generation, Settlements and Billing extracts posted on 11/12

– Notice Date: 11/14/07 – 11/15/07

– Market Notice: M-C111407-02

– Root Cause: • Complications with the batch scheduling software encountered while

completing an ERCOT requested rerun of the 11/8 batch (10/31 trade date) resulted in the 11/8 batch running on Sunday 11/10 and completing on 11/11. We moved forward with the 11/9 batch and it completed at approximately 5:00 p.m. on Saturday. When Lodestar batch ran out of order, the controls table was not accurately populated.

– Resolution: • Control table rows previously manually entered did not take into account

that the 11/8 batch day data was not complete until 11/11. Related to market notice M-A111207-02 . A SIR has been requested to modify the control record creation process.

10/29/07 - 11/26/07 Extract Issues

• Issue:  Delays with posting several data extracts/reports– Notice Date: 11/21/07 - present– Market Notice: M-A112107-03– Root Cause:

• Extracts are posting one day behind due to batch run delays.

– Resolution: • Currently analyzing several key areas including correcting an issue

with backups and tuning database performance.