ERCOT Lodestar Extracts Timing Recommendation
ERCOT
May 24, 2005
Overview
Lodestar Extracts Current Extract Process for Lodestar Extracts from Data Archive ERCOT Recommended Process for Lodestar Extracts from ODS ERCOT Recommended Timing Process for Lodestar Extracts from ODS
Lodestar Extracts
SCR 727 ESIID Extract Resource ID Extract (New) Load Extract Generation Extract Settlements & Billing Extract Settlement Input Data Extract (New) Shadow Pricing Extract Public Reference Table Extract (New)
SCR 727 ESIID Extracts Current Process from Data Archive
Data Type: ESIID level data
Data Flow Process: At 5:00AM, Lodestar Data Archive process runs to capture the existing records with addtimes/timestamps of previous day 4:00AM through current day 3:59:59AM
Frequency: Generation and posting of extract occurs daily and is posted to market participants after being audited
Extract Creation Trigger: Automatically by data inserts/updates/deletes
Load, Generation, and Settlements & Billing Extracts Current Process from Data Archive
Data Type:– MOS and bill determinant trade date (Settlement Input Data) data
as early as two days after operating day and may receive data changes up until the initial settlement
– Data Aggregation and Settlement interval data outputs
Data Flow Process: At 5:00AM, Lodestar Data Archive process runs to capture the existing records with addtimes/timestamps of previous day 4:00AM through current day 3:59:59AM
Frequency: Extract process occurs Tuesday – Saturday only and is modified as necessary for holidays
Extract Creation Trigger: By market participants scheduling the extract; Not dependent on Data Archive Process
Load, Generation, and Settlements & Billing Extracts Current Process from Data Archive – con’t
Known Reasons for Improvement:– Potential for trade date outputs to be contained in multiple extracts– No dependency on settlement trade date approvals allowing trade
date outputs to be received in multiple extracts with different addtimes/timestamps to account for backouts/reruns of unapproved settlements prior to posting an approved statement
– Inconsistent synchronization of SCR 727 ESIID Extract data in conjunction with corresponding Load & Generation Extract postings
Shadow Pricing ExtractsCurrent Process from Data Archive
Data Type: Contains shadow pricing data
Data Flow Process: At 5:00AM, Lodestar Data Archive process runs to capture the existing records with addtimes/timestamps of previous day 4:00AM through current day 3:59:59AM
Frequency: Extract process occurs Tuesday – Saturday only and is modified as necessary for holidays
Extract Creation Trigger: Automatically by data inserts
SCR 727 ESIID Extracts and Resource ID Extracts ERCOT Recommended Process from ODS
Data Type: – SCR 727 ESIID Extract contains ESIID level data (granular data)– Resource ID Extract contains Resource ID level data (granular
data) - New
Data Flow Process: Continuous data flow from the Lodestar Source System -> Replicated Source System -> Lodestar ODS capturing intraday data updates and point in time selectivity
Frequency: Generation and posting of extract occurs daily and is posted to market participants after being audited
Extract Creation Trigger: Automatically by data inserts/updates/deletes
Recommended Changes:– Data load time window for the granular data Point in Time is
captured by the initiation of granular data changes (Lodestar Batch usage data loading) at SYSDATE -3 to the next day’s initiation of granular data changes (Lodestar Batch usage data loading)
– Produce on a 2 day lag from the data load time window stop time to accommodate for extract trade date synchronization and Lodestar ODS data synchronization including scheduled & unscheduled environment maintenance and troubleshooting data & system issues
– Data included within daily extract is synchronized to one posting of a complete batch of trade dates for the Load, Generation and Settlements & Billing Extracts
Timing proposal discussed with DEWG at February - May 2005 meetings
SCR 727 ESIID Extracts and Resource ID Extracts ERCOT Recommended Process from ODS – con’t
SCR 727 ESIID Extract and Resource ID Extract Timing Change
Day Extract is Generated and
Published
CurrentData Addtime Time Window
RecommendationData Addtime Time Window
Monday Sunday 4:00AM - Monday 3:59:59AM Friday ~4:00:00 AM – Saturday ~3:59:59 AM
Tuesday Monday 4:00AM - Tuesday 3:59:59AM Saturday ~4:00:00 AM – Sunday ~3:59:59 AM
Wednesday Tuesday 4:00AM - Wednesday 3:59:59AM Sunday ~4:00:00 AM – Monday ~3:59:59 AM
Thursday Wednesday 4:00AM - Thursday 3:59:59AM Monday ~4:00:00 AM – Tuesday ~3:59:59 AM
Friday Thursday 4:00AM - Friday 3:59:59AM Tuesday ~4:00:00 AM – Wednesday ~3:59:59 AM
Saturday Friday 4:00AM - Saturday 3:59:59AM Wednesday ~4:00:00 AM – Thursday ~3:59:59 AM
Sunday Saturday 4:00AM - Sunday 3:59:59AM Thursday ~4:00:00 AM – Friday ~3:59:59 AM
Load, Generation, and Settlements & Billing Extracts ERCOT Recommended Process from ODS
Data Type: – Load and Generation Extracts contain all outputs for a trade date in
one extract– Settlement & Billing Daily Extract contains Settlements input data
used and output data created for each approved trade date for all trade dates within a batch run becoming a complete trade date specific extract
– Provide complete batch run with all Trade Dates approved accounting for subsequent Trade Date reruns prior to Settlements Approval
Data Process Flow: Continuous data flow from the Lodestar Source System -> Replicated Source System -> Lodestar ODS capturing intraday data updates and point in time selectivity
Load, Generation, and Settlements & Billing Extracts ERCOT Recommended Process from ODS – con’t
Frequency: Generation and posting of extract occurs daily and is posted to market participants after being audited
Extract Creation Trigger: By market participants scheduling the extract
Recommended Changes:– With extract creation and posting delay, No additional logic
necessary to accommodate for receipt of unapproved Trade Date data and extract schedule changes
– Up to an additional 3 day lag to the delivery timeline to accommodate for synchronization of extracts with approved settlement data (weekends & holidays), Lodestar ODS data synchronization including scheduled & unscheduled environment maintenance and troubleshooting data & system issues
Recommended Changes:– Consistent synchronization of SCR 727 ESIID Extract data and RID
Extract data in conjunction with corresponding Load & Generation Extract postings – Granular data necessary to shadow settle is posted 2 days prior to the Aggregation and Settlement extracts
Timing proposal discussed with DEWG at May 2005 meeting
Load, Generation, and Settlements & Billing Extracts ERCOT Recommended Process from ODS – con’t
Load, Generation, and Settlements & Billing Extracts Timing Change
Day Extract is Produced and
Published
CurrentData Addtime Time Window
Batch Trade Dates for due Settlement Statements
RecommendationData Time Window
Monday No extract produced and published Thursday, Friday, Saturday Wednesday Batch Trade Dates
Tuesday Friday 12:30 AM - Monday 12:29:59AM(Fri, Sat, Sun batch)
Sunday Thursday Batch Trade Dates
Wednesday Monday 12:30AM - Tuesday 12:29:59AM(Mon batch)
Monday Friday Batch Trade Dates
Thursday Tuesday 12:30AM - Wednesday 12:29:59AM(Tues batch)
Tuesday Saturday Batch Trade Dates
Friday Wednesday 12:30AM - Thursday 12:29:59AM(Wed batch)
Wednesday Sunday Batch Trade Dates
Saturday Thursday 12:30AM - Friday 12:29:59AM(Thurs batch)
No approved statements posted
Monday Batch Trade Dates
Sunday No extract produced and published No approved statements posted
Tuesday Batch Trade Dates
Settlement Input Data Extracts (New)ERCOT Recommended Process from ODS
Data Type: Contains all input data passed from MOS to the settlement system or entered by settlement personnel; data that is currently provided in the Settlement & Billing Extracts
Data Flow Process: Continuous data flow from the Lodestar Source System -> Replicated Source System -> Lodestar ODS capturing intraday data updates and point in time selectivity
Frequency: Generation and posting of extract occurs daily and is posted to market participants after being audited
Extract Creation Trigger: By market participants scheduling the extract
Settlement Input Data Extracts (New)ERCOT Recommended Process from ODS – con’t
Recommended Changes:– Redefines Settlement & Billing Extract to only provide complete
approved trade date inputs and outputs synchronized with the Load and Generation Extracts
– Data load time window is captured by the initiation of granular data changes (Lodestar Batch usage data loading) at SYSDATE -3 to the next day’s initiation of granular data changes (Lodestar Batch usage data loading)
Settlement Input Data Extracts Timing Change
Day Extract is Produced and
Published
CurrentSettlements & Billing Extract Data Addtime Time Window
RecommendationNew Extract
Data Addtime Time Window
Monday No extract produced and published Friday ~4:00:00 AM – Saturday ~3:59:59 AM
Tuesday Friday 12:30 AM - Monday 12:29:59AM(Fri, Sat, Sun batch)
Saturday ~4:00:00 AM – Sunday ~3:59:59 AM
Wednesday Monday 12:30AM - Tuesday 12:29:59AM(Mon batch)
Sunday ~4:00:00 AM – Monday ~3:59:59 AM
Thursday Tuesday 12:30AM - Wednesday 12:29:59AM(Tues batch)
Monday ~4:00:00 AM – Tuesday ~3:59:59 AM
Friday Wednesday 12:30AM - Thursday 12:29:59AM(Wed batch)
Tuesday ~4:00:00 AM – Wednesday ~3:59:59 AM
Saturday Thursday 12:30AM - Friday 12:29:59AM(Thurs batch)
Wednesday ~4:00:00 AM – Thursday ~3:59:59 AM
Sunday No extract produced and published Thursday ~4:00:00 AM – Friday ~3:59:59 AM
Shadow Pricing ExtractsERCOT Recommended Process from ODS
Data Type: Contains shadow pricing data
Data Flow Process: Continuous data flow from the Lodestar Source System -> Replicated Source System -> Lodestar ODS capturing intraday data updates and point in time selectivity
Frequency: Generation and posting of extract occurs daily and is posted to market participants after being audited
Extract Creation Trigger: Automatically by data inserts
Recommended Changes:– Data load time window is captured by the initiation of granular data
changes (Lodestar Batch usage data loading) at SYSDATE -3 to the next day’s initiation of granular data changes (Lodestar Batch usage data loading)
Settlement Input Data Extracts Timing
Day Extract is Produced and
Published
CurrentData Addtime Time Window
RecommendationData Addtime Time Window
Monday No extract produced and published Friday ~4:00:00 AM – Saturday ~3:59:59 AM
Tuesday Friday 12:30 AM - Monday 12:29:59AM(Fri, Sat, Sun batch)
Saturday ~4:00:00 AM – Sunday ~3:59:59 AM
Wednesday Monday 12:30AM - Tuesday 12:29:59AM(Mon batch)
Sunday ~4:00:00 AM – Monday ~3:59:59 AM
Thursday Tuesday 12:30AM - Wednesday 12:29:59AM(Tues batch)
Monday ~4:00:00 AM – Tuesday ~3:59:59 AM
Friday Wednesday 12:30AM - Thursday 12:29:59AM(Wed batch)
Tuesday ~4:00:00 AM – Wednesday ~3:59:59 AM
Saturday Thursday 12:30AM - Friday 12:29:59AM(Thurs batch)
Wednesday ~4:00:00 AM – Thursday ~3:59:59 AM
Sunday No extract produced and published Thursday ~4:00:00 AM – Friday ~3:59:59 AM
Public Reference Table Extracts (New)ERCOT Recommended Process from ODS
Data Type: Contains full tables of non-interval market level data (ex. TDSP, STATION, CMZONE, etc.) from Lodestar extracts
Data Flow Process: Continuous data flow from the Lodestar Source System -> Replicated Source System -> Lodestar ODS capturing intraday data updates and point in time selectivity
Frequency: Generation and posting of extract occurs daily and is posted to market participants after being audited; only one daily extract available (no historical postings); delivery timeline same as SCR 727 ESIID and Resource ID Extracts
Extract Creation Trigger: Automatically by data inserts/updates/deletes
What’s Next?
Comments, Feedback, Questions
ERCOT complete Analytical Designs of Execution Phase for the following projects:
– PR 30026 – Resource ID Extract – PR 40007 – SCR 727 Phase II (SCR 727 ESIID Extract and Public
Reference Table Extract)– PR 40042_06 – Lodestar Batch Extracts (Load, Generation,
Settlements & Billing, Settlement Input Data, and Shadow Pricing Extracts)