MNDOT

Traffic Monitoring and Analysis Business Processes

Contents subject to change

Office of Transportation Data and Analysis, Traffic Forecast and Analysis Section

11/28/2011 version

PURPOSEThis document was prepared as part of a User Requirements assessment (see Project 1237) that is intended to support decisions to replace existing data systems and business methods with updated and better coordinated software, application and business processes. It is also the basis for developing a ‘living document’ that will evolve as changes are made to the traffic monitoring program, even as alternatives to replace systems and processes are considered. Selected staff of the Traffic Forecasting and Analysis Section (part of the Office of Transportation Data and Analysis) has been involved in the authoring and review of the contents. The document is organized into four basic sections:

1.0 Introduction and Context – This section contains a brief historical summary of developments over the past thirty years that created the systems and processes that Minnesota Department of Transportation (MNDOT) now uses.

2.0 General Traffic Monitoring Program Precepts – This section presents ‘Business Rules’ that continue to guide decisions about change. The section also contains a Business Terminology glossary (incomplete).

3.0 Business Process Diagrams and References to Supporting Documentation – This section contains business process flow diagrams that illustrated sequential linkages between business partners, processes, internal and external data systems, data sets, and critical decision points. Each diagram is or will be supported by electronic file documents that refer directly to sections of the diagram. Copies of these documents in their current state (dated 2/28/2011) are currently stored on the TDA Office network drive N: under the folder: \TrafMon\UserNeeds with folders named according to what type of data is documented (volume, classification or WIM, then folders for short duration, continuous or annual process). The document names contain the diagram letters and numbers that correspond to the first, ‘overview’ diagram. This section will eventually contain a schedule that documents when certain process are performed for any given count cycle. However, such a schedule must be read with an understanding that two to three count cycles overlap each other creating an approximate 30 month work cycle timeframe.

4.0 Detailed Data System Documentation for Mature Data Systems – This section will be augmented with database structure, object descriptions, variable definitions, and listings/purpose descriptions for scripts as time allows and as the need arises.

1.0 INTRODUCTION AND CONTEXT

The logically related series of systems and processes that support MNDOT’s Traffic Monitoring Program perform a vital role in traffic data collection, quality control, analysis and reporting for the department. Over time, personal computer and server technology has been used to replace many of the paper, pen and pencil methods, and at the time, innovative mainframe systems that were used in the past. Most of the replacement or new systems were designed and programmed by MNDOT analysts using software tools, programming skills and electronic ‘infrastructures’ that were available. Each system has one or more of the following components:

1. Traffic sample scheduling and cycle administration2. Location and attribute management (add, modify, retire, delete) using logically embedded or

manually implemented business rules3. Data load utilities supporting various sources of external data

4. Raw data or aggregate data quality assurance and quality control algorithms and analyst tools5. Graphical and non-graphical data representation6. Reporting tools to assist the analyst and to provide information for internal and external users7. Outside system and subsystem interface utilities based upon key field equivalencies (typically using

file exports) to facilitate data exchange, interoperability, and production of final map, report and GIS products.

During the 1970s and 1980s systems to support continuous volume data and short duration vehicle classification data relied upon mainframe programs that were either programmed separately in COBOL or were closely tied to the department’s Transportation Information System, still being used today. Considerable manual data entry was required for these systems. Processing of short duration volume counts and mapping of count locations, Annual Average Daily Traffic (AADT) and Heavy Commercial Average Daily Traffic (HCADT) was done by hand.

During the 1990’s greater use of personal computer and server based computer aided drafting (CAD) software and files, database management systems and statistical analysis software led to less reliance upon the mainframe and manual processing but increased the variety of skills and coordination required within MNDOT to implement such change. These years saw a) the implementation of SAS programs to manage the continuous volume data and create temporal adjustment factors, b) the use of automated mapping using a server based database and coordinated CAD files, c) development and implementation of an expert system to screen and impute continuous volume data, d) the use of detector data from the Regional Traffic Management Center (RTMC), e) the use of relational database systems to manage short duration volume and vehicle classification data, and f) a rapid expansion of the weigh-in-motion program requiring the use of vendor provided analysis software, auto-calibration subroutines and development of an ORACLE database to facilitate post data collection ‘data salvaging’.

The first decade of the century saw a) the development of new heavy commercial average daily traffic (HCADT) estimation algorithms and data base and spreadsheet applications to support such estimation, b) an increased automation of data flows for RTMC detector data and partner agency and district short duration counts, c) the use of edit flags in continuous volume data to filter out vehicle classification data prior to federal submission, d) an increase in the use of geographic information systems (GIS) given the initiative to create a statewide base map, and e) the addition of software supplied by Dr. Kwon at the University of Minnesota, Duluth (UMD) to process and display WIM data and an increasing amount of continuous vehicle classification data. WIM data auto-calibration and data ‘salvaging’ was not being done during this time. Even now, enhancements to existing systems are being implemented and other useful spreadsheet and database applications are being created by MNDOT staff to support new processing needs and analysis requests. Additionally, efforts are being made to better coordinate field locations across the various programs so duplication of effort and travel can be avoided.

Omitted from this history are the databases and spreadsheets developed to assist in forecasting traffic due to the necessity of focusing on the Traffic Monitoring Program.

All systems developed since 1995, except the WIM Oracle database, exist today. Traffic Monitoring Program business rules have also changed over the years as computerization has increased the quantity and quality of data, improved the administration of programs, led to more transparency, and provided a springboard for additional data sharing and dissemination. It is expected that as this user needs analysis proceeds, business rules and replacement data systems will evolve together. To that end the next

section outlines many of the current, general business rules that underlay the traffic monitoring program so evolution may begin again.

2.0 GENERAL TRAFFIC MONITORING PROGRAM PRECEPTSMNDOT’s Office of Transportation Data and Analysis (TDA) provides estimates of AADT on all public roadways in the state each year following recommended practices. Additionally, estimates of vehicle classification are made for all state highways and continuous vehicle/axle weights are collected. These data support federal reporting, roadway design, safety analysis and other planning and budgeting requirements. Operational data to support traffic signals or ramp meters are not directly managed by TDA but provide a source of volume data once screened and aggregated.

Systematically addressing client needs in the context of limited resources has led to a structured method for providing the ‘right’ amount of data at the ‘right’ times. For example, hourly count data is not routinely collected or stored for short duration volume counts since there is very little call for it from stakeholders. That does not mean, however, that such data collection wouldn’t be helpful for other reasons.

Throughout the traffic monitoring program the concepts of count location (points), traffic segments (two-way roadway segments) and sampling cycles provide the conceptual underpinnings for many of the processes and systems contained in this document. It is assumed that there is variability in everything measured and that variability can mask trends when considering longer time frames. Most processes illustrated in this document attempt to accommodate or mitigate the effects of that variability. The statements that follow represent current practice and policies that guide decisions about change within the traffic monitoring program.

1. All roadways need AADT estimates every year (and Trunk Highways need HCADT every year) and should represent ‘typical conditions’ as best as possible when estimated during a count year.

2. AADT estimates consist of five different types. Each type has different statistical properties and appropriate uses. Counts from each count program should supplement the count record on a segment for other programs where possible and appropriate.

a) Default values based upon sampling and applied to ‘uncounted’ route systems,

b) Annually adjusted prior year estimates (for growth),

c) Calculated based upon adjacent or contributing segment AADTs,

d) Short Duration counts taken on 2, 4, 6 or 12 year cycles using a mix of counting equipment during the temperate time of year (April – October). Recounts are taken where necessary. Counts are immediately adjusted for day-of-week and month using factors from prior years (by averaging up to three years’ factors), and axle correction factors when applicable (TH only). When determining AADT, adjusted counts are tempered with past estimates and adjacent counts. The official count cycle length may change for count locations, but under limited and controlled circumstances.

e) Permanently installed sensors at continuous counting locations (RTMC, continuous volume and classification at ATRs and WIMs).

3. AADT estimates apply to roadway segments on the ‘counted’ system using traffic segment delineation criteria:

a) Sliding-scale longitudinal AADT change along a segment based on tolerances for ‘along roadway’ variation in traffic volumes. Special counts taken simultaneously may provide data supporting alternate segment definitions.

b) TH intersections with TH roadways create traffic segment breaks.

c) HPMS sample segments spur the need for a counted segment if not regularly sampled

d) Ramps require counts.

4. All traffic segments have ‘direct’ or ‘indirect’ traffic count locations determined by traffic monitoring program requirements (continuous vs. short duration weekday sampling, volume vs. classification, WIM needs) and field considerations (accessibility, safety, available equipment, anchorage points, partner agency needs). When possible, the entire historical record of when and where counts were taken, as stored in databases and geographic representations, should be preserved.

5. HCADT and other vehicle classification estimates vary by AADT segment for the TH system and are based upon :

a) Short-duration vehicle classification counts adjusted for day-of-week, month and AADT estimate when counted. These consist of approximately 1200 ‘parent’ sites distributed across the entire TH system and sampled on a six year cycle. Recounts are taken where necessary.

b) Continuous vehicle classification data (additional ‘parent sites’ and source of data to calculate vehicle classification daily and seasonal factors).

c) Relationship between short duration vehicle classification counts (parent sites) and related contiguous segments (children segments). A child segment HCADT estimate is usually based upon the actual parent site HCADT estimate, and an addition or subtraction of an HCADT increment based on the difference between the parent and child AADTs and one of two proportions of HCADT present in that difference (depending upon an urban or rural distinction).

d) Annually adjusted prior year estimates (for growth).

6. We maintain that traffic data of all varieties are accurate unless recounts, specific calibration data, serious departures from historical data, or field checks indicate that additional modification or qualification of the data is necessary.

7. Comments from field staff and outside agencies can affect the ultimate determination of a traffic estimate. Such comments are routinely solicited from County and Municipal State Aid partner agencies and evaluated in the context of past counts, local land use and roadway network changes.

8. Functional class is not used as a predictor for axle correction or temporal adjustment factor application due to the wide variety of traffic patterns within any functional classification.

9. The traffic volume program attempts to collect data at 100% of the count locations for all segments in a jurisdiction in the year(s) within a count cycle for roadways of the following types: Trunk Highways, County State Aid, Municipal State Aid (and County Roads in the Minneapolis/St. Paul metropolitan area).

10. Weigh-in-Motion data are collected by lane and used to establish ESAL factors for use in pavement design related forecasts, and to alert enforcement officials. Calibration is performed twice a year and average front axle weights and GVW distributions of Type 9 trucks are used to track calibration drift in the data.

11. Traffic data collection equipment and locations are treated as infrastructure when incorporated into the department’s GIS. The management of such locations is independent of route or other attributes since a given data collection point may exist on the same roadway over time with varying route designations and characteristics.

12. All traffic data collection, processing and adjustment procedures require a full audit trail. Methods are usually documented and transparent to data users on-line and upon demand.

13. Data about the performance of data collection devices is captured and analyzed since such data permit a closer evaluation of the accuracy and reliability of the measurements and subsequent estimates. Data review by an analyst or ‘intelligent’ data processing system will initiate a series of field verification activities or bench tests to check for bias and other data related performance issues.

14. Continuous volume data are edited or ‘imputed’ when bias is introduced by known equipment malfunctions and when data are missing. Original data are always retained. Such flagged edits are based upon hourly, directional, day-of-week and month-of-year averages and ranges calculated for that site from the past year. The analyst may modify estimates to accommodate week within month variability, events, holidays, and other special considerations.

15. When necessary, traffic data are qualified using codes, comments and other means to guide decisions about subsequent data uses. For example, data flags on imputed volume data prevent the reporting of vehicle classification data from the same site and time period to the federal government, yet the imputed volume data is usually included in the computation of adjustment factors used to adjust short duration volume counts.

16. For all traffic monitoring program related data there should be only one source for official annual estimates that are used for federal reporting and other internal/external uses. Extreme efforts are continually being made to synchronize traffic data systems with the most appropriate roadway representations. It follows that historical traffic estimates need to be illustrated with historically accurate roadway representations.

17. Annual estimates of AADT, HCADT, other vehicle classification breakdowns, and K and D factors are usually produced in a coordinated fashion as soon after February as possible following the data collection year. Depending upon available staffing and workload this data release has occurred between April and the beginning of June.

Glossary of Terminology – not completeAADT - Annual Average Daily TrafficAASHTO - American Association of State Highway and Transportation OfficialsATR - Automatic traffic recorder – Continuous Volume and/or ClassificationDD - Directional distribution of trafficDHV - Design hour volumeDOW - Day of weekESALs - Equivalent single axle loadsFHWA -Federal Highway AdministrationGIS - Geographic Information System GVW Gross vehicle weightHCADT or HCAADT - Annual Average Daily Truck TrafficHPMS - Highway Performance Monitoring SystemK factor - (design-hour traffic as a percent of AADT – Federal reporting uses 30 th highest hour)MADT - Monthly Average Daily TrafficMEPDG - Mechanistic-Empirical Pavement Design GuideQA/QC - Quality assurance / quality controlSAS - Statistical Analysis Software (produced by the SAS Institute)TDA - Transportation Data and Analysis office in MNDOTTIS - Transportation Information SystemTMG - Traffic Monitoring Guide produced by the FHWAVMT - Vehicle-miles of travelWIM - Weigh-in-motion

3.0 BUSINESS PROCESS DIAGRAMS AND DOCUMENTATION

“Existing Conditions” Traffic Monitoring Business Process Diagrams

The objective of this activity was to obtain and summarize a complete set of business process flow diagrams describing the current Traffic Monitoring program from the MNDOT central office point of view. Temporal aspects of the process flows are addressed in the diagrams as well. It should be noted that some processes are recursive and involve feedback loops that require a more complete understanding of time-related dependencies.

Each process within a group of related processes are illustrated with a unique color that represents the predominant role the process plays according to the terms below. The MNDOT traffic monitoring program has been diagrammed and described using the following terms:

1. Collection – (PINK) Prefaced with the letter ‘A’ - Polling, gathering and administering the collection of continuous, short-duration and weigh-in-motion (WIM) data.

2. Processing, Analysis and Storage – (BLUE) Prefaced with the letter ‘B’ for daily/weekly or (GREEN) ‘C’ for monthly - These processes include processing data from counters, entering and evaluating data, program status review, creating draft estimates of AADT and monthly reporting.

3. Year-end or Annual Processing – (YELLOW) Prefaced with the letter ‘D’ - Developing adjustment factors - monthly, daily, hourly, design-hour volume( DHV), directional distribution, axle correction, and . estimation of Annual Average Daily Traffic (AADT), Heavy Commercial Annual Average Traffic (HCADT), Vehicle Weight characteristics and annual Vehicle Miles Traveled (VMT).

4. Reporting – How customer needs are met through electronic data transfers, hard copy or web tools and reports.

5. Determine Traffic Monitoring Program Traffic Segment and Count Locations – Prefaced with the letter ‘L’ – Continually assessing the need to determine where traffic segment breaks need to be created, merged, or deleted, and where count locations need to added, modified, recoded or retired. Count location attributes are maintained in multiple data systems and GIS layers creating a synchronization problem between systems.

To be more meaningful for the Traffic Monitoring Program staff, selected names of files or process execution files were included on process shapes. More complete representations of process mechanics exist in the supporting documents found on the network drive N: as noted in the introductory ‘Purpose’ section but are incomplete at this time.

Shapes used in the diagrams

Drums represent mature data systems or database applications.

Rectangles represent an analyst initiated processes.

Diamonds represent Key Decision points.

Parallelograms represent interim data files or sources of ‘outside’ information.

‘Wave Bottomed’ Rectangles represent printed or electronic reports.

‘Slant Roofed’ Rectangles represent manual data entry processes.

Please see the Process and Data Flow Diagrams later in this document.

Double click on the Adobe Document reference – requires Adobe Reader.

Thoughts About the Future

Anticipating the future, the traffic monitoring staff has expressed interest in the additional areas outlined below. These are not prioritized and do not necessarily include additional needs and desires of external users. Project 1237 will help to identify how outside users and partners view the program and how those views may affect future traffic monitoring program activities.

1. Unifying and better coordinating count location designation and attribute coding so all counting programs reference a centralized or shared location data structure (preferably using a GIS interface).

2. Incorporate more data screening algorithms and post polling calibration adjustments to vehicle weight data from WIM installations.

3. Maintain or increase control over data system and application development to increase flexibility when responding to changing customer requirements, program scheduling and methodological improvements.

4. Increase the ability of the volume count program to accept hourly data.5. Increase the use of the internet to collect data from partner agencies or outside data services.6. Increase the use of GIS in data screening processes, analysis, reporting and mapping.7. Increase the number of sources and types of data that can be accessed using the future system.8. Improve the ability to create reports and disseminate data to outside interests from all aspects

of the traffic monitoring program.

Adobe (.pdf) Document containing Process and Data Flow diagrams, pages 1-15. Double-click on diagram below to see all diagrams.(Page 17 is simply a diagram assembled together from pages 1-15.)

4.0 ‘Mature’ Data Systems – (detailed data system documentation available if needed)

COUNTS – Relational Database Management System using client/server R:BASE tables, objects, application and scripts

VCCOUNTS – Database using ACCESS, application, VB scripts, linked EXCEL spreadsheets, macros

RTMC Data Warehouse – Custom Program (Kwon) using Microsoft Visual Studio and .Net Framework 3.5 to work with archived sensor data files

SAS Continuous Data System – SAS datasets and program scripts

Expert System for Continuous Volume Data Screening and Editing – Nexpert Object for Windows (1995), and a structured file system

Parent/Child Spreadsheet/Database System – Applies segment specific classification counts to related up/downstream traffic segments

Bull Piezo – Custom Program (Kwon ‘09) for continuous vehicle class graphing, charting, reports, analysis for seasonal adjustment factors (Kwon)

Bull Guide – Custom Program (Kwon ‘09) for visualizing and evaluating WIM data for Load Spectra and for creating input files for MEPDG software

Bull Reports – Custom Program (Kwon ‘09) for graphing and reporting class and weight data, and evaluating WIM data for Load Spectra

Peak Converter – Custom Program (Kwon ‘09) for reading vendor binary format files, converting them to spreadsheet friendly .csv format, and providing a ‘standard’ directory structure for data storage

Spatial Data Warehouse – ESRI, ORACLE and managed enterprise (MNDOT) resource

Kwon – Professor Taek Kwon, Transportation Research Laboratory, University of Minnesota, Duluth