PennDOT ETL (PETL) · PennDOT ETL v1.0 User Guide . 2.0 Steps 2.1 Step 0: Preparing your Spreadsheet Data File Before you even start PETL, you need a spreadsheet data file. Your spreadsheet

PennDOT ETL (PETL) Version 1.0

User Guide

January 21, 2015

PennDOT ETL v1.0 User Guide

Document Revision Information Version Num. Edit Date Edited By Comments 1.0 July 17, 2013 GeoDecisions Initial Draft

2

Table of Contents


Table of Contents

Document Revision Information.................................................................................. 2 1.0 Introduction .............................................................................................. 4

1.1 Purpose ................................................................................................................ 4 1.2 Background ........................................................................................................ 4 1.3 Typical Workflow ................................................................................................. 5

2.0 Steps ......................................................................................................... 6 2.1 Step 0: Preparing your Spreadsheet Data File ............................................... 6

2.1.1 Defined Columns .......................................................................................... 6 2.1.2 Generic Columns .......................................................................................... 6 2.1.3 File Examples: ................................................................................................ 7

2.1.3.1 RMS Points Example ............................................................................... 7 2.1.3.2 RMS Lines Example ................................................................................. 7 2.1.3.3 Lat Long File Example ............................................................................ 8

2.2 Step 1: Upload Data File ................................................................................... 8 2.2.1 Launching PETL from a Mapping Application ......................................... 8 2.2.2 Launching Petl from the Stand-Alone App .............................................. 9 2.2.3 Starting Row Number ................................................................................... 9 2.2.4 Error Uploading File ....................................................................................... 9

2.3 Step 2: Project Creation / Column Mapping ............................................... 10 2.3.1 Project Creation .......................................................................................... 10 2.3.2 Start Row ...................................................................................................... 10 2.3.3 Column Mapping ....................................................................................... 10

2.4 Step 3: Review Validation / Promote to Live ............................................... 12 2.4.1 Error Details .................................................................................................. 13

2.5 Step 4: Promoted to Live ................................................................................. 14 2.5.1 Promoted from within a Host Mapping Application ............................. 14 2.5.2 Promoted from the Stand Alone Application ........................................ 15

3.0 Scenarios ................................................................................................ 16 4.0 FAQ ......................................................................................................... 16

4.1.1 Can you delete a project after it has been created? ......................... 16 4.1.2 Can a project file contain points and lines? .......................................... 16 4.1.3 Can you append data to existing project that has already been promoted to live? .................................................................................................... 16 4.1.4 How does the automatic column mapping work? .............................. 16

5.0 Application Support .............................................................................. 17 6.0 Glossary.................................................................................................. 17

3

Table of Contents


1.0 Introduction 1.1 Purpose This document explains to end-users how to use the PennDOT ETL application.

1.2 Background PennDOT ETL is a browser based application which allows users to import CSV or Excel spreadsheets containing linear or point geometry data into a GIS ready database. PETL replaces the current desktop GIS2000 application, but provides a huge gain in features and time savings. It allows users to display files directly on a map by dragging/dropping the file onto the map (i.e. in GIS-IQ, MPMS-IQ, and Maintenance-IQ). It also improves the ease of mapping fields by allowing users to drag/drop database fields onto the imported data file’s columns. PETL auto-detects header rows and automatically matches database fields using the file’s column headers.

4

Table of Contents


1.3 Typical Workflow The following flow diagram illustrates the typical workflow of using the PETL application as described above.

Screen 1: Upload Spreadsheet File

Screen 2: Project Creation/ Column Mapping

Screen 3: Review Results/ Promote to Live

Fix errors in file / Start over

Screen 4: Data Promoted

Open Project in Mapping Application

MPMS IQ Maintenance IQ

Start a new File

5

Table of Contents


2.0 Steps 2.1 Step 0: Preparing your Spreadsheet Data File Before you even start PETL, you need a spreadsheet data file. Your spreadsheet data file is a collection of rows and/or lines. The first row in the spreadsheet should be a header row with column titles, although this is not required. Each row after the header should contain geometry and descriptive data about each point or line. The geometry can be RMS points, RMS lines, or latitude/longitude points.

PETL can accept Excel files (.xls or .xlsx) and CSV (Comma Delimited Files).

If the row contains both RMS columns and Lat/long data, it will use the RMS data and ignore the Lat/Long data. 2.1.1 Defined Columns

Column Name Size RMS Point Required

RMS Line Required

Lat/Long Required

County 2 Yes Yes

State Route 4 Yes Yes

Segment Begin 4 Yes Yes

Offset Begin Number Yes Yes

Segment End 4 Yes

Offset End Number Yes

Jurisdiction 1

Side Indicator 1

Latitude Number Yes

Longitude Number Yes

2.1.2 Generic Columns

PETL allows you to associate 15 additional columns to be included with the mapping data. This way you can see more descriptive data when viewing the points/lines on a map. You can include up to 5 character, 5 number, and, 5 date fields. Your spreadsheet can contain many more columns, but you are

6

Table of Contents

http://www.dot.state.pa.us/Internet/Bureaus/pdBOMO.nsf/infoRMRIRMS


limited to including these 15 generic fields when promoting your data to the ArcGIS database. 2.1.3 File Examples:

The following screenshots illustrate a few typical examples of PETL spreadsheets. 2.1.3.1 RMS Points Example

2.1.3.2 RMS Lines Example

7

Table of Contents


2.1.3.3 Lat Long File Example

2.2 Step 1: Upload Data File 2.2.1 Launching PETL from a Mapping Application

If you are in a mapping Web application such as Maintenance IQ or MPMS IQ, simply drag your spreadsheet data file onto the map. You will see a shaded overlay appear on the screen as well as a “Drop CSV or Excel File here” message.

8

Table of Contents


2.2.2 Launching PETL from the Stand-Alone App

If you are using the stand-alone application, the first screen you see will contain a welcome screen with a gray box. Simply drag your spreadsheet data file onto the grey box.

2.2.3 Starting Row Number

The start row is the first row in the spreadsheet where there is actual data. Most of the time, you do NOT need to enter a value. When a file is uploaded, PETL automatically detects the correct start row for data in the spreadsheet.

2.2.4 Error Uploading File

If you receive the error message “The application is having trouble opening this Excel file…” then open the spreadsheet in Excel. Select File, Save As, Change the Save as Type to CSV (Comma Delimited), click Save, Ok, Yes. Then try uploading the newly saved .csv file. PETL should be able to process all types of Excel files, but it is possible there is an unexpected compatibility issue. The failsafe option is to convert the spreadsheet to a CSV.

9

Table of Contents


2.3 Step 2: Project Creation / Column Mapping 2.3.1 Project Creation

After your spreadsheet is uploaded, you arrive at the Column Mapping screen.

• You can create a new project or update an existing project.

• New projects must have unique names.

• When updating a project, the existing project data on stage is overwritten.

• Check Append to add your data to the existing project rather than overwriting the existing data.

2.3.2 Start Row

As mentioned previously, PETL usually is able to detect the correct start row. Most files have one header row and then data immediately following. So the start row is two in most cases. If PETL does not select the correct start row, click Start Over and manually set the correct start row in Step 1. 2.3.3 Column Mapping

This program automatically maps or matches the database field names with the columns in your imported spreadsheet. This automatic matching is done based on header names. The automatic matching is not always correct, especially if

10

Table of Contents


the spreadsheet headers are cryptic or missing. You have a chance here to manually correct and modify the field mappings.

• To map database fields to a spreadsheet column, simply drag the appropriate field from the unused database list and drop it onto the appropriate column header.

• To remove a mapped field from a column, drag the field from the column header and drop it onto another field header or a field name in the unused list.

• When dropping fields, be sure to move your mouse pointer into the box.

• When finished, click Validate and Submit to Stage. This Validate button is only visible when you have entered a valid project name and have mapped enough fields to create points or lines.

11

Table of Contents


2.4 Step 3: Review Validation / Promote to Live After clicking Validate, PETL validates your data moves it into stage tables. When valuation is finished you will see the Results screen in Step 3.

Here is what each of the values mean:

Validated No problems with the data in these rows

Warnings The rows have issues that will cause them to fail. These rows will not be promoted.

Errors

The rows have issues that will cause them to fail. These rows will not be promoted.

Empty Rows Rows that are empty in the spreadsheet

Rows Processed

The total number of rows processed. (Not including header rows.)

Promotable Rows

Rows that can be promoted to production for this file. (Sum of validated and warnings rows.)

12

Table of Contents


Here is what the buttons do:

• Promote Validated Data moves your project data to the "live" tables so that the points and lines can be accessed by mapping applications. If you have no "promotable" rows, this button will not be visible.

• Start a New File allows you to go back to the beginning to import a new data file. You can then add more data to this project or create another project.

• Go Back to Field Mapping allows you to go back to adjust the field/column mappings. Maybe a field was not mapping correctly and is causing a ton of errors?

Trash this Project completely deletes what you just did and deletes your project. You cannot delete projects that have already been promoted to live.

2.4.1 Error Details

Click Show Error Details to view a table displaying why certain rows failed. The Row Number matches the row number shown in Excel.

13

Table of Contents


2.5 Step 4: Promoted to Live Congrats! You promoted your spreadsheet data to the live database. What happens next depends on if you activated PETL from a mapping application or from the “stand-alone” application. 2.5.1 Promoted from within a Host Mapping Application

If you launched PETL from a “host” mapping application, then the host application should load the project you just promoted onto the map. Simply click the Close PETL button and see your project on the map.

14

Table of Contents


2.5.2 Promoted from the Stand Alone Application

If you promoted your data from the PETL stand-alone application, you have the option of viewing your project from a list of mapping applications shown. The links shown should open the mapping application in a new browser window and include the project Id in the URL so that the project can be loaded onto the map.

In this example, you notice the green lines of this Indiana county based project.

15

Table of Contents


3.0 Scenarios In Step 2, you have to switch to Advanced Mode to create and name scenarios. Using scenarios is similar to appending data to an existing project. A scenario allows you upload multiple files for one project. You can create and name a scenario for each file uploaded. You can also replace and revalidate a particular scenario without affecting the other scenarios in the project. Scenarios only exist within the context of validating data in stage. Once you are done and happy with validating the scenarios, you promote the project to live. When the data is promoted to live, all the scenarios are grouped together as one project.

4.0 FAQ The following is a list of Frequently Asked questions and facts that are not completely self-evident. 4.1.1 Can you delete a project after it has been created?

When you create a new project in Step 2 and validate the data, you have the option in Step 3 to “trash” or delete the project before it gets to live. However, once you promote a project to “live” you lose the ability to delete the project from the PETL tool. You do have the option to overwrite an existing project with new data and rename the project. 4.1.2 Can a project file contain points and lines?

Yes. PETL will check to see if enough data is available to map a line. If not, it will next check to see if it can map a point. So you can include points and lines in one file. 4.1.3 Can you append data to existing project that has already been promoted to

live?

No. You only append data to an existing project when you are creating and validating a project in stage. Once you promote a project to “live” all the data gets promoted to together and one group. If you want to add data to an existing project, you will need to create a new separate project, or reload all the data again for the existing project. The mapping applications typically allow you to view more than one project Id at once. 4.1.4 How does the automatic column mapping work?

The PETL application checks the column headers in the spreadsheet to do the automatic mapping. It does not check the actual data. The PETL application has a list of aliases for the various column database names. For example, for County, it looks for the words like county, cty_code, and cty. For most files the column mapping just works, but it all depends on the column headers in the

16

Table of Contents


spreadsheet. If the spreadsheet has no header row, no automatic column mapping occurs.

5.0 Application Support If you have an issue with PETL that this document does not address, the following is a list of personnel that may be able to assist.

JR “BOB” Grugen GIS Supervisor and PETL power user.

Patrick Kielty GIS Analyst and PETL user

Jeff Fennimore Product Manager and Systems Adminstrator

Jim Lawruk GeoDecisions software developer

Nate Reck GeoDecisions Architect

6.0 Glossary Supply a glossary of all terms and abbreviations used in this document.

AJAX – In reference to PETL, the application uses the browser to make “AJAX” HTTP Requests and responses to send and receive data behind the scenes without a full page reload.

BOTS – “Bunch of Tools” A collection of map related tools that can be easily included and integrated into a JavaScript based Web application.

CRUD – (Create, retrieve, update, delete) Short way of encompassing all data related tasks. You can 1) create or insert data, 2) retrieve or query data, 3) update or edit data, and 4) delete or remove data. CSV – A comma delimited spreadsheet file with the file extension (.csv)

GIS2000 – The old application that PETL is replacing.

PennBots – Collection of tools to be integrated into Web applications. The tools include a front-end UI component along with back-end services.

PennDOT – Pennsylvania Department of Transportation

PETL – The application this document describes which passes spreadsheet data into an ArcGIS ready database.

17

Table of Contents


Project – A collection of geometry data and associated descriptive data tied to a user. A project has an id, name, and can contain a collection of files/scenarios.

Scenario – A term for dividing a project into a set of uploaded data files. A scenario represents a subset of project data. When a file is uploaded, it represents one scenario. A file upload can represent a new scenario or can overwrite an existing scenario. A project can contain multiple scenarios, but most simple projects only have one scenario.

SPA – (Single Page Application) An application that runs in the browser that does not require any post backs or URL changes. Refreshing the browser resets the application. PETL is a SPA.

18

Table of Contents

Documents

PennDOT ETL (PETL) · PennDOT ETL v1.0 User Guide . 2.0 Steps 2.1 Step 0: Preparing your Spreadsheet Data File Before you even start PETL, you need a spreadsheet data file. Your spreadsheet