Steward system Seokjoo Lee University of Florida [email protected] [email protected]

Steward system

Seokjoo Lee

University of Florida

[email protected]

Table of contents

System configuration Operation Management and expansion Migration Hotter program

http://cdwserver.ce.ufl.edu/steward/index.html

top_top.asp

left_menu.asp

common_top_menu.asp

main.asp

main.asp

overview.asp

resources.html

maps.html

default.asp

index_main/TSS_Facility_Level_Reports/

TSS_Facility_Level_Reports.asp

index_main/TSS_Section_Level_Reports/

TSS_Section_Level_Reports.asp

index_main/TSS_Station_Level_Reports/

TSS_Station_Level_Reports.asp

index_main/TVT_Station_Level_Reports/

TVT_Station_Level_Reports.asp

index_main/TVT_ETL_Reports/report.asp

index_main/csv_reports/csv_reports.asp

main.asp

maps/TSS_interactive_map.html

maps/d2_interactive_map.html



Text Links (7)

Document Links (1)

Presentation Links (2)

Archive Links (8)

Spreadsheet Links (3)

top_top.asp

left_menu.asp

common_top_menu.asp

main.asp

Script (client-side) Links (7)


Generic Links (100)


http://cdwserver.ce.ufl.edu/steward/index_main/

TSS_ETL_Reports/report.asp


Generic Links (100)

Steward System configuration

InternetFTP server

Univ of Florida

D2 TMCSunGuide Data Archive



Data backup2 x 1TB external data storages1

DB & (Application backup)500GB local data storage2

Steward DB server & web server

1. As of 2-26-08, district data size is 360GB. (D2: 160GB, D4: 94GB, D6: 83GB, FTP backup: 18GB) 2. As of 2-26-08, DB size is 72GB and application backup is 108GB.

Steward data flow

FTP server Steward DB server & web server

TSS data from each

district

Backup and uncompress

Transformation (SunETLUtility)Assign Station/Lane IDBasic quality checkAggregation

Load into DB and update views

User access from web

Every day, TSS data are delivered from D2, D4 and D6 into Steward. Data sizes are 100MB, 230MB and 110MB each.

Steward system configuration-ETL(1)

FTP server- Filezila FTP server version 0.9.30 beta version is installed to serve the

TSS data transfer from each district. - C:\Steward_ftp is assigned for the FTP storage- Each district can access C:\Steward_ftp\DistrctX folder for its file

transfer. - Host IP: cdw880.ce.ufl.edu Administrator User ID: cdw Administrator Password: trc513 District User ID: cdw_user_d1 ~ cdw_user_d8 District Password: steward08 Data backup- 2BrightSpark SyncBack version 3.2.19.0 is used to backup FTP folders

into local file storage daily- Two copies of backup are created into two local file storages (external

hard drive: 1GByte each, USB interface)

Steward system configuration-ETL(2)

Data transformation- Custom-build SunETLUtility is used to transform the raw TSS

data into the aggregated Steward data format- SunETLUtility can be executed in command-line or stand-alone

program. - SunETLUtility needs the following subfolders: FacilityDataFolder: Input facility data folder ToConvertFolder: Input TSS data folder ConvertedDataFolder: Output folder for converted data/report GroupDataFolder: Output folder for group data- SunETLUtility command line format: SunETLUtility /dDx /iInputDataFolder /oOutputDataFolder " example) SunETLUtility /d"D2" /i"c:\D2Data" /o"c:\Steward\Input"

Steward system configuration-DB(1)

Steward database tools- Oracle Database 10g 10.2.0.1.0 Enterprise edition- Oracle Warehouse Builder 10g 10.2.0.1.311: GUI interface- Oracle Workflow 2.6.4.0.0: Scheduler

Steward database/web server- Dell Optiplex GX620- Pentium 4 3.2GHz/ 4GB Ram/Windows XP3 SP3- Seagate 160GB HDD (SATA I) for Oracle SW- Seagate 500GB HDD (SATA II) for database

1Oracle patch version 10.2.0.3 is expected to be used. metalink


Steward DB tables- Design Strategy for Steward database -- Keep the simple architecture to avoid the costly joins -- Allow duplicate data tables/views rather than on-the-fly computations

- Fact tables (Cubes) -- Station-level tables: TSS_5min, TSS_15min and TSS_1hr -- Lane-level tables: TSS_5min_lane, TSS_15min_lane and TSS_1hr_lane

- Dimension tables (Dimension) -- date_ref, time_ref, tss_station


- Materialized views: -- SYSTEM_LANE_5MIN, SYSTEM_LANE_15MIN, SYSTEM_LANE_1HR -- VOLUME_MAP_5MIN, VOLUME_MAP_15MIN, VOLUME_MAP_1HR -- TRAFFIC_COUNTS_5MIN, TRAFFIC_COUNTS_15MIN,

TRAFFIC_COUNTS_1HR -- PM_5MIN, TT_REL_5MIN -- MAX_FLOW_5MIN, MAX_FLOW_15MIN, MAX_FLOW_1HR -- EFF_DET_LEN_5MIN, EFF_DET_LEN_15MIN, EFF_DET_LEN_1HR

- Materialized views are partitioned by Date and District and indexed by Station_ID, Hour and Date

- Fact tables will be partitioned and indexed on next major revision- Materialized views are refreshed on demand. They will be

updated into on-commit on next major revision

DATE_REF

PK DIMENSION_KEY

DAY_OF_CAL_MONTH DAY_OF_CAL_QUARTER JULIAN_DATE DAY_END_DATE DAY_NAME DAY_ID DAY_OF_CAL_WEEK DAY_DAY_CODE DAY_START_DATE DAY_DESCRIPTION DAY DAY_OF_CAL_YEAR DAY_TIME_SPAN MONTH_OF_QUARTER CALENDAR_MONTH_CAL_MONTH_CODE CALENDAR_MONTH_DESCRIPTION CALENDAR_MONTH_START_DATE CALENDAR_MONTH_TIME_SPAN CALENDAR_MONTH_ID CALENDAR_MONTH_NAME CAL_MONTH_NUMBER MONTH_OF_YEAR CALENDAR_MONTH_END_DATE CALENDAR_QUART_CAL_QUARTER_CO CALENDAR_QUARTER_NAME CALENDAR_QUARTER_TIME_SPAN QUARTER_OF_YEAR CALENDAR_QUARTER_START_DATE CAL_QUARTER_NUMBER CALENDAR_QUARTER_ID CALENDAR_QUARTER_DESCRIPTION CALENDAR_QUARTER_END_DATE CALENDAR_YEAR_NAME CALENDAR_YEAR_ID CAL_YEAR_NUMBER CALENDAR_YEAR_DESCRIPTION CALENDAR_YEAR_CAL_YEAR_CODE CALENDAR_YEAR_END_DATE CALENDAR_YEAR_START_DATE CALENDAR_YEAR_TIME_SPAN

TIME_REF

PK DIMENSION_KEY

DESCRIPTION TIME ID

TSS_STATION

PK DIMENSION_KEY

NUM_OF_LANES STATION_ID DESCRIPTION ROADWAY_ID DETECTOR_TYPE DETECTOR_UNIT UPNODE STATE_MILEPOST ID ROAD LANE_CAPACITY SPEED_LIMIT LONGITUDE ROADWAY_MILEPOST STATUS LATITUDE COUNTY COUNT_STATION

TSS_15MIN

FWY_SPD FWY_VOL FWY_OCC SPD_CV VOL_RATIO SPD_RATIO ENTRY_VOL EXIT_VOL FWY_QA ENTRY_QA EXIT_QA HOV_SPD HOV_VOL HOV_OCC HOV_QAFK1,I1 DATE_REFFK2,I2 TIME_REFFK3,I3 TSS_STATION

TSS_5MIN


TSS_1HR


TSS_5MIN_LANE

LANE1_SPD LANE1_VOL LANE1_OCC LANE1_OBS LANE2_SPD LANE2_VOL LANE2_OCC LANE2_OBS LANE3_SPD LANE3_VOL LANE3_OCC LANE3_OBS LANE4_SPD LANE4_VOL LANE4_OCC LANE4_OBS LANE5_SPD LANE5_VOL LANE5_OCC LANE5_OBS LANE6_SPD LANE6_VOL LANE6_OCC LANE6_OBS ONRAMP1_SPD ONRAMP1_VOL ONRAMP1_OCC ONRAMP1_OBS ONRAMP2_SPD ONRAMP2_VOL ONRAMP2_OCC ONRAMP2_OBS ONRAMP3_SPD ONRAMP3_VOL ONRAMP3_OCC ONRAMP3_OBS OFFRAMP1_SPD OFFRAMP1_VOL OFFRAMP1_OCC OFFRAMP1_OBS OFFRAMP2_SPD OFFRAMP2_VOL OFFRAMP2_OCC OFFRAMP2_OBS OFFRAMP3_SPD OFFRAMP3_VOL OFFRAMP3_OCC OFFRAMP3_OBSFK1,I1 DATE_REFFK2,I2 TIME_REFFK3,I3 TSS_STATION

TSS_1HR_LANE


TSS_15MIN_LANE


DATE_REF

PK DIMENSION_KEY

DAY_OF_CAL_MONTH DAY_OF_CAL_QUARTER JULIAN_DATE DAY_END_DATE DAY_NAME DAY_ID DAY_OF_CAL_WEEK DAY_DAY_CODE DAY_START_DATE DAY_DESCRIPTION DAY DAY_OF_CAL_YEAR DAY_TIME_SPAN MONTH_OF_QUARTER CALENDAR_MONTH_CAL_MONTH_CODE CALENDAR_MONTH_DESCRIPTION CALENDAR_MONTH_START_DATE CALENDAR_MONTH_TIME_SPAN CALENDAR_MONTH_ID CALENDAR_MONTH_NAME CAL_MONTH_NUMBER MONTH_OF_YEAR CALENDAR_MONTH_END_DATE CALENDAR_QUART_CAL_QUARTER_CO CALENDAR_QUARTER_NAME CALENDAR_QUARTER_TIME_SPAN QUARTER_OF_YEAR CALENDAR_QUARTER_START_DATE CAL_QUARTER_NUMBER CALENDAR_QUARTER_ID CALENDAR_QUARTER_DESCRIPTION CALENDAR_QUARTER_END_DATE CALENDAR_YEAR_NAME CALENDAR_YEAR_ID CAL_YEAR_NUMBER CALENDAR_YEAR_DESCRIPTION CALENDAR_YEAR_CAL_YEAR_CODE CALENDAR_YEAR_END_DATE CALENDAR_YEAR_START_DATE CALENDAR_YEAR_TIME_SPAN

TIME_REF

PK DIMENSION_KEY

DESCRIPTION TIME ID

TSS_STATION

PK DIMENSION_KEY

NUM_OF_LANES STATION_ID DESCRIPTION ROADWAY_ID DETECTOR_TYPE DETECTOR_UNIT UPNODE STATE_MILEPOST ID ROAD LANE_CAPACITY SPEED_LIMIT LONGITUDE ROADWAY_MILEPOST STATUS LATITUDE COUNTY COUNT_STATION

VOLUME_MAP_15MIN

I3 DATE1 TIMEI2 HOUR DISTRICT FACILITYI1 STATION_ID DIRECTION STATION_MP UPNODE_ID ENTRY_VOLUME FWY_VOLUME EXIT_VOLUME LINK_INPUT LINK_OUTPUT DIFFERENCE PCNT_DIFF

TRAFFIC_COUNTS_5MIN

I3 DATE1 TIMEI2 HOUR DISTRICT FACILITYI1 STATION_ID NUM_OF_LANES DIRECTION STATION_DESC STATION_MP TOTAL LANE1_VOL LANE2_VOL LANE3_VOL LANE4_VOL LANE5_VOL LANE6_VOL BALANCE FWY_QA ON_RAMP1 ON_RAMP2 ON_RAMP3 ON_RAMP_QA OFF_RAMP1 OFF_RAMP2 OFF_RAMP3 OFF_RAMP_QA COUNTY COUNT_STATION

TRAFFIC_COUNTS_1HR

DATE1 TIMEI2 HOURI3 DISTRICT FACILITYI1 STATION_ID NUM_OF_LANES DIRECTION STATION_DESC STATION_MP TOTAL LANE1_VOL LANE2_VOL LANE3_VOL LANE4_VOL LANE5_VOL LANE6_VOL BALANCE FWY_QA ON_RAMP1 ON_RAMP2 ON_RAMP3 ON_RAMP_QA OFF_RAMP1 OFF_RAMP2 OFF_RAMP3 OFF_RAMP_QA COUNTY COUNT_STATION

TRAFFIC_COUNTS_15MIN


SYSTEM_LANE_1HR

I3 DATE1 TIMEI2 HOUR DISTRICT FACILITYI1 STATION_ID DIRECTION STATION_DESC STATION_MP LANE1_VOL LANE1_SPD LANE1_OCC LANE2_VOL LANE2_SPD LANE2_OCC LANE3_VOL LANE3_SPD LANE3_OCC LANE4_VOL LANE4_SPD LANE4_OCC LANE5_VOL LANE5_SPD LANE5_OCC LANE6_VOL LANE6_SPD LANE6_OCC ONRAMP1_VOL ONRAMP1_SPD ONRAMP1_OCC ONRAMP2_VOL ONRAMP2_SPD ONRAMP2_OCC ONRAMP3_VOL ONRAMP3_SPD ONRAMP3_OCC OFFRAMP1_VOL OFFRAMP1_SPD OFFRAMP1_OCC OFFRAMP2_VOL OFFRAMP2_SPD OFFRAMP2_OCC OFFRAMP3_VOL OFFRAMP3_SPD OFFRAMP3_OCC

VOLUME_MAP_5MIN


VOLUME_MAP_1HR


SYSTEM_LANE_5MIN


SYSTEM_LANE_15MIN


PM_5MIN

I3 DATE1 TIMEI2 HOURI1 STATION_ID DISTRICT DIRECTION STATION_DESC UPNODE_ID UPNODE_DESC STATION_MP LENGTH LINK_VOL NUM_OF_LANES VOL_PER_LANE VEH_MI VEH_HR SPEED DELAY KENERGY DENSITY VOL_TO_CAP

TT_REL_5MIN

I3 DATE1 TIMEI2 HOURI1 STATION_ID DISTRICT DIRECTION STATION_DESC STATION_MP UPNODE_ID UPNODE_DESC UPNODE_MP LENGTH SPEED_LIMIT LINK_VOL NUM_OF_LANES AV_SPEED AV_TT TT_INDEX PCNT_ONTIME ONTIME_DELAY CONGESTION_DELAY

TRAFFIC_COUNTS_5MIN


TRAFFIC_COUNTS_1HR

DATE1 TIMEI2 HOURI3 DISTRICT FACILITYI1 STATION_ID NUM_OF_LANES DIRECTION STATION_DESC STATION_MP TOTAL LANE1_VOL LANE2_VOL LANE3_VOL LANE4_VOL LANE5_VOL LANE6_VOL BALANCE FWY_QA ON_RAMP1 ON_RAMP2 ON_RAMP3 ON_RAMP_QA OFF_RAMP1 OFF_RAMP2 OFF_RAMP3 OFF_RAMP_QA COUNTY COUNT_STATION

TRAFFIC_COUNTS_15MIN


MAX_FLOW_1HR

I2 DATE1 DISTRICT FACILITYI1 STATION_ID DIRECTION STATION_DESC STATION_MP LANE_NUM MAX_FLOW MAX_TIME MAX_GAP

EFF_DET_LEN_15MIN

I3 DATE1 TIMEI2 HOUR DISTRICT FACILITYI1 STATION_ID DIRECTION STATION_DESC STATION_MP LANE1_VOL LANE1_SPD LANE1_OCC LANE1_EFF_DET_LENGTH LANE2_VOL LANE2_SPD LANE2_OCC LANE2_EFF_DET_LENGTH LANE3_VOL LANE3_SPD LANE3_OCC LANE3_EFF_DET_LENGTH LANE4_VOL LANE4_SPD LANE4_OCC LANE4_EFF_DET_LENGTH LANE5_VOL LANE5_SPD LANE5_OCC LANE5_EFF_DET_LENGTH LANE6_VOL LANE6_SPD LANE6_OCC LANE6_EFF_DET_LENGTH

MAX_FLOW_5MIN


EFF_DET_LEN_1HR


EFF_DET_LEN_5MIN


MAX_FLOW_15MIN


Steward system configuration-Web(1)

Web design concepts - Minimize the on-the-fly computations by using pre-

developed materialized views - Minimize the complex queries on the client side by

using the query functions where the materialized are not applicable.

Steward web tools - Microsoft Internet Information Services 5.1 for web

server - Text-based ASP, JavaScript, HTML


Steward web architecture- Home- Overview- Resources- Maps- District data -- TSS Facility-Level Reports -- TSS Section-Level Reports -- TSS Station-Level Reports -- TSS ETL Reports -- TVT Station-Level Reports -- TVT ETL Reports -- Report Archive


Steward web pages

Steward operations -ETL process (1)

D2, D4 and D6 pushes archived TSS data into Steward FTP server daily at 3AM, 2AM and 2AM.

- Average file sizes are 15MB, 30MB and 15MB in compressed file format (100MB, 230MB and 110MB in uncompressed format)

- File location: c:\Steward_FTP\DistrictX

File backup and uncompressing - Backup location: u:\DISTRICT_DATA\Districtx\rawdata - Uncompressing location: c:\Program Files\TRC\SunETLUtility\ToConvert

Data transformation - Raw traffic data are processed using custom program (SunETLUtility) - Process the data files -- Assign Station/Lane ID -- Aggregated 1min, 5min, 15min and 60min lane/station data -- Run the basic quality check

Steward operations -ETL process (2) -- One input data file would generate one converted file, one daily report, eight

aggregated data file. For example, TSS-03032009--1.dat from District 2 would generate the following output files

ConvertedData\D2_TSS-03032009-0.csv ConvertedData\D2_TSS-03032009-DailyReport.Log GroupData\D2_TSS-03032009-1S.csv GroupData\D2_TSS-03032009-1L.csv GroupData\D2_TSS-03032009-5S.csv GroupData\D2_TSS-03032009-5L.csv GroupData\D2_TSS-03032009-15S.csv GroupData\D2_TSS-03032009-15L.csv GroupData\D2_TSS-03032009-60S.csv GroupData\D2_TSS-03032009-60L.csv

- Report process results to Steward Admin and users - Archive all the processed files and prepare the data loading -- Backup location: u:\DISTRICT_DATA\Districtx\processeddata\ConvertedData u:\DISTRICT_DATA\Districtx\processeddata\GroupData

Data backup - Steward_FTP folder are backup into U drive at 4AM daily - U drive are back-up into the identical V drive at 5AM daily


Data loading - Oracle Process Flow modules are used to load TSS data files - For one raw TSS file, six processed data files are loaded (5min,

15min, 60min station and lane data) - Oracle Process Flow modules can be executable manually using

Control Center or automatically using Oracle Schedules


After data uploading, data changes can be verified from their table row counts.


Database performance can be monitored from Oracle Enterprise Manager

Steward operations - Current status

Traffic data from districts

*: Daily data are not received, data CRC error or our ftp server issues.

District TSS Data Range Missing days*

Roadways Data issues

2 6-28-07 to 2-11-09

6 days I-95 I-295

4 5-1-08 to 2-1-09 14 days I-95, I-595 I-95 NB has two HOT lanes

118 New stations are added at I-75

6 8-1-08 to 2-1-09 7 days SR-826, I-195, I-75

I-95 data are not stabilized yet.

Steward operations -Web

Steward web statistics

Management plan

Daily data loading

- Daily transform report by email DB performance

- Oracle enterprise manager console Web statistics

- WebLog Expert Lite version 5.6

Expansion scenario (1)

New data for CDW: three possible scenarios

Limited information on district changes/information - New HOT lanes in District 4 and 6 - New facility (I-75) in District 4 - Limited information (I-295 milepost) in District 2


Expansion might cause the performance issues on preprocessing, DB performance or web interface.

Unexpected system configurations could impact the system modifications:

- Two HOT lanes in District 4 and District 6

- HOT lanes in NB are operational but SB will be implemented later

- 30sec aggregation interval in D5


New analysis report

- New materialized views needs be created.

- Web interface needs be revised

Statewide traffic CDW at other states

Rob Hranac at Berkeley Transportation Systems defines the progress of Archived Data User Service (ADUS) as follows:

- Data Reports (for decision support system) Application (web 2.0) Prediction Control Automation

Most of states are in the stage 2 (Report) and move toward stage 3 (Application)

PeMS at California are developing integration of traffic data and incident data on GIS.

PeMS are trying to integrate the transportation simulation model with the traffic data

WisTransPortal integrated traffic data database with lane closure system

U of Maryland developed traffic data warehouse for Washigton DC, Maryland and Virginia

U of Maryland developed the web-based analytical tools for the traffic data and incident data

Migration plan

Web service - Web sources are server independent and ready to

migrate Database - Migrate Oracle warehouse builder metadata - TSS data will be re-loaded rather than table migration - TSS data tables will be reconfigured for Index and

partitions - Refresh method for TSS materialized views will be

reconfigured from on-demand to on-commit ETL process - ETL process will be customized for new system

Hotter program

Utility program for analyzing the operation of High Occupancy Toll (HOT) lanes within SunGuide traffic management systems in Florida.

I-95 in D4 and D6 has HOV lanes on SB and HOT lanes on NB

Input data are generated from Steward manually Hotter program generates analysis report on HOV/HOT lanes

in comma-delimited (CSV) file format Results may be plotted if desired from the analysis files

Documents

Steward system Seokjoo Lee University of Florida [email protected] [email protected]