ATLAS

Erich Borchardt (2009) Minnesota Department of Agriculture Steven Crisp (2005 – 2010) Michigan State University

ATLAS - Flow of Data

The flow of data from field to database usually begins by downloading CSV files from a GPS unit. This involves clicking one button on the GPS tab marked "Get Waypoints". This will download and prompt the user to save the file in the correct location all in one step. If coordinates outside of the continental United States are encountered a warning dialog will appear to advise you that your data may not be accurate as this tool was not designed to be used outside of the STS project area. It is advisable to use unique names for every CSV that you manage. When files are uploaded they are stored in an original files location and the copy in the downloads folder is moved. In case you forget a name, the program will warn that you are about to overwrite previous data and allow you to cancel or proceed. Once you have saved a csv in the downloads folder you are ready for the next step.

Clicking the CSV button will begin the process of importing and spatially validating your trap data against a list of intended sites. This also is a one click operation. The data will be quickly evaluated as to what state it intersects and if it happens that the data lie within two ore more states, you will be asked to choose which one to validate in. This process crawls over every site in the entire state with a fast algorithm that only spends time in regions where clusters of waypoints exist. This way a file containing data from widely separated geographic regions can be processed all at once.

The program operates on the assumption that every non-9000 trap should belong with a node and the closer the trap is to a node the greater the likelihood that it belongs to it. In areas of uncertainty where grid spacing changes to a delimit and multiple target buffers intersect, the distance out values will be compared for a recommendation. Correct placements where no conflict exists, will appear in blue. If the program encountered an area of uncertainty, the results will be drawn in red. The results can be analyzed with the navigation tools and the identity tool. If it is necessary, incorrect trap data can be edited in the attribute table.

The attribute table is launched by clicking on the table button. When the table is first opened it automatically validates the integrity of the trap data string. It will check for every parameter and condition that can be encountered and advise you of any errors it finds with a note next to the record and highlighting the record in red. To fix those areas edit mode is set to allow edits and the cursor is placed in a cell, then appropriate corrections are made. Each cell that is edited will then be highlighted in green to show you where you made corrections. When you have finished, it will be necessary to run the validations again from the tools menu. If everything was alright in the trap data column, there shouldn't be any more highlighted records. At this point you should commit your edits or lock the edit mode and save changes. Then under the file menu simply choose save and the file will be put into queue for FTP transfer. Note that if it was not successfully validated, it will not allow you to perform this step. Other fields like line numbers, dates, and altitudes are forced into correct format if needed and should not normally need editing. The program will not attempt to validate those fields, it only formats them to ensure upload.

The final step in getting the information to the database is the actual upload. After you have configured the program to reflect your state and contact information this is again a one button process. Files will appear in the various boxes of the FTP tab according to what their status is. All files in the Files Ready for Upload box will be sent when the Upload button is depressed. The results of the transfer will then be reported to you. If it failed for some reason, then nothing will have been sent and a dialog will advise of the reason why. When transfer is successful, the files will move to the Files Successfully Transferred box and will be grouped together by the date which they were sent. They can be opened locally on your machine by double clicking on them, but they are only for reference at this point.

2

ATLAS Interface – A quick tour

These names presented below will be used to describe the Atlas interface throughout the manual.

The bulk of the Atlas interface is split between the Table of Contents (TOC) and the Map Area. The TOC is where the geographic layers of data can be managed. The Map Area is the working portion of the display

The Menu item list is used in almost any computer application. Click on a menu item to active the pop-up menu of options.

Clicking on the Task tabs is like flipping between pages. When the MAP tab is active the Map Area shows geographic features. When the GPS Management tab is active controls relevant to downloading the GPS are visible. The FTP tab will not be used.

Status

Status messages

Map Area

Progress bar

Task tabs

Menu Floating toolbar

Table of Contents (TOC)

Status messages are brief notifications that the program is performing a task what stage it is at in a task or the result of a task. For example the program might be "searching" for a connected GPS unit then "parsing" data until a final tally or records processed will be displayed.

The Progress bar is used to indicate activity during a process and approximately what percentage of a given task has been performed. It may stop from time to time and that alone can not indicate a hung process.

3

ATLAS Step by Step Guide

to Data Retrieval and Validation

5

ATLAS Step by Step Guide to Data Retrieval and Validation

You will learn how to open Atlas, load a pre-made project with data layers already added and symbolized, connect and download a GPS unit, review the GPS records for accuracy and make edits if necessary. First, launch Atlas from the desktop shortcut: Opening an Atlas Project A project has been set up and saved with useful layers symbolized and labeled. This project can be found at C:\GM_STS\ with the file name Atlas2009.atls

Select the File menu item and Open Project. In the Open Project window browse to C:\GM_STS\ select Atlas2009.atls and press Open.

This project has two layer groups. The Data Layers group contains information useful for finding your location on a large scale, routes, counties, quads (labeled with the abbreviation).

The second group (auto named Group_2) contains Minnesota roads divided into 5 tiles (NW, NE, SW, SE and Metro, just turn on the ones you need), lakes and other information you may find useful while zoomed in and looking at individual sites.

Adding additional data layers Additional geographic data layers may be added by pressing the Add Layer button

Data layers are located in C:\GM_STS\Shapefiles, Minnesota specific layers under the MN_Data folder, layers covering the entire STS project area are under atlas_data folder. Layers available include Boundary Water Canoe Area, Camp Ripley or State Forest trails.

6

Downloading the GPS

Plug the GPS download cable into the serial port at the back of the laptop and connect the cable to the GPS and turn on the GPS unit.

2

3

1

1- Click the GPS Management Tab

2- When downloading the Garmin V be sure to check the box to filter out preload points. Skipping this step will allow all preload points will be included in your CSV, which is bad. It is best to always check this box.

3- Click Get Waypoints to initiate the GPS Download

Program will extract waypoints from GPS unit

The Save CSV File As window opens, enter the name of the download file.

The file should be named using the trappers’ initials and the date of the download, ex. GMU0516.csv. The date used in file-naming must not precede dates found in trapping records as it will generate errors at the database level. If the trapper has two GPS units (or a gadget and a Garmin V) for download, use consecutive dates ex. GMU0514.csv.*

Atlas prompts you to enter a file name and saves downloads in the C:\GM_STS\GPS_Files\Downloads folder.

Now it is time to look at the data and once you have confirmed the data is complete (and free of preload points) clear the GPS unit.

*Note: Each trapper’s initials are loaded into the database at the beginning of the season, the database searches the GPS download file names and will generate an error if the initials are not correct. If you start downloading Robert’s GPS with the initials RMB but call him Bob, you may not change to BMB once the season has started. In the event of trappers with duplicate initials, a set of initials will be assigned.

7

Loading GPS data

The long version: Now begins the process of importing and spatially validating your trap data against a list of pre-determined sites. The data will be assessed as to what state it intersects and you will be asked to choose which one to validate in. The program cross checks every site in the state with an algorithm that only spends time in regions where clusters of waypoints exist and a file containing data from widely separated geographic regions can be processed all at once.

The short version: Simply put, stuff happens and you will be able to see where trapper data doesn’t match what the internal database is expecting.

The Open CSV file for processing button allows you to load a downloaded GPS file for validation and viewing. Press the button, select the file you will validate and press Open.

Atlas may ask which state you would like to validate, choose no for any except Minnesota.

Spatial validation begins and progress may be monitored at the bottom of the Atlas display. When completed click on the Map tab to view the data you have just downloaded.

Loading the GPS data also loads the base sites file entitled MN_Sites. This data layer contains every site to be placed in Minnesota and is confusing at first. Turn off this layer by clicking the box with a check mark in the Table of Contents.

With the target sites turned off you can view the GPS points more easily.

Still, it may be a good idea to change the symbol of the mn_sites to ease confusion when comparing GPS to sites. Double click mn_sites in the TOC. A box titled “In the Display Properties of: mn_sites” opens. Click Point Color button and choose a new color (I like purple) and click OK. The color palette closes and returns you to the Display Properties window.

You may change the shape of the point symbol as well by selecting from the radio buttons under the point color button. When finished making changes click the X in the upper right corner of Display Properties to close the window.

8

Validating GPS data

With the GPS data is loaded you may use the zoom tool to take a closer look. Click the zoom tool and you will notice the cursor turns to a magnifier glass when hovering in the map frame. Click and drag the cursor to produce a zoom frame, release the mouse button and the map will zoom the box you have just created. Zoom

During the spatial validation process Atlas has added a color code to each point. Correct placements will appear in blue. If the program encountered an area of uncertainty, the results will be drawn in red. Problematic data can be viewed with the navigation and identity tool and incorrect trap data can be edited in the attribute table.

Some text can be seen as you zoom in. Zoom in a bit more to see what Atlas is suggesting.

Atlas is unable to determine which target to associate with this site. Labeling the GPS points may help.

A red circle indicates a point which does not match the target site.

To label the GPS points double click the CSV file name to open the Display Properties dialog. Select the Labels tab and in the pull down select Trap Data. Now press the Create Labels button. Close Display Properties to return to the map. The GPS points will be labeled.

The point does not have a proper trap number. The trapper will need to provide additional information before edits can be made. When you are ready to make an edit, open the attribute table.

Note: It is helpful to write down information on questionable traps as you perform the visual check.

9

Editing in the Attribute Table

The attribute table is launched by clicking on the table button.

When the table is first opened it automatically validates the integrity of the comment string. It will advise you of any errors it finds with a note next to the record. In the example below, the trapper failed to enter trap type.

.

In order to make the necessary changes to the TRAP_DATA field the table needs to be made editable.

Set table to edit mode by selecting Edit and Allow Edits

Place cursor in the cell, and make appropriate corrections. In this case, there should be a D (for Delta trap) entered at the end of the comment string. After an edit is made the cell will be highlighted in green.

10

ONLY EDIT THE TRAP_DATA FIELD!

Once all edits are completed, it will be necessary to run the validations again from the tools menu.

When all highlighted records have been addressed, commit your edits or lock the edit mode and save changes.

Finally, under the File menu select Save to CSV

Upon successfully saving, the file is moved from Downloads folder and is saved to C:\GM_STS\GPS_FILES\QUEUE.

Atlas will not allow you to save the file until all the STS Validation errors are corrected, you will be given an error message stating which line contains the invalid entry. Go back and edit that entry and try saving again. Because of this, it is a good idea not to begin edits until you are sure they may all be addressed.

11

Deleting a Record

Sometimes it will be necessary to delete a record from the table. To accomplish this select the record and replace the # symbol in the BOOKMARK field (furthest to the left on the table) with a capitol X.

Altas will warn you that the row “will be excluded from the final csv.”

Filtering the Records

It is possible to filter the records in the table to make it easier to view. Select from Tools the Filter Records option. The Table Query Filter window.

In the Table Query Filter window, choose an attribute you would like to use to filter the data. Click on it to select it.

Once you select an attribute the Unique Values window populates with options.

In the example, the Error_Code has been selected.

The Unique Values box populates with values from the ERROR_CODE field.

Now click the expression in the middle.

Click Apply

The table sorts to the records which meets the criteria.

12

The Track Log

The Garmin V automatically records the location of the GPS unit every few moments like a cookie crumb trail. This trail may be downloaded and viewed in Atlas.

Select Get Tracks from the GPS menu. The main view area switches to the GPS tab and the download begins, this may take a few moments.

After the download you will be asked to name the file.

Use the same naming convention as downloads, initials and date. If there isn’t a folder already, create one under GPS_Files called TrackLogs. The tracks are saved as a shapefile which may be viewed in Altas by adding the track file as a layer.

To view the tracklog, switch to the Map tab. Add a layer and browse to the TrackLogs folder, select the file.

The tracklog layer will be added.

Opening the file outside of Atlas Notepad

You may be interested in finding out more about a certain trapper’s routine especially if they are falling behind. Opening the CSV in notepad allows you to look at the time each trap is placed to see when they are starting and ending their day. You can also quickly figure out the average time the trapper takes between traps. This can help in pin-pointing troubles the trapper may be having.

You can review the file using Notepad to make a quick scan for inconsistencies. You may make edits to the CSV text file in Notepad and save them without affecting the data integrity.

Microsoft EXCEL

While you can quickly scan the data in Notepad, the data may be out of order. You may open a file using MS Excel to sort the data, however you must NEVER SAVE THE FILE using MS Excel. Doing so results in terrible things happening to the data which are time consuming and boring to fix. So, don’t do it.

For this reason it is best never to view the csv files in Excel.

Clearing the GPS Unit Once the GPS data is downloaded the GPS will need to be cleared. Never leave the points loaded onto the GPS once they have been downloaded.

a) Press MENU twice to access the main menu. b) Use the ROCKER KEYPAD to select FIND and hit ENTER c) When the Find pop-up menu appears select ‘Waypoints’ and hit ENTER d) In the window select Waypoints by Name, hit ENTER e) Press PAGE to exit the onscreen keyboard and view the waypoint list f) Press MENU to bring up the options for the waypoint list g) Select Delete All, press ENTER h) Choose Yes, press ENTER i) Hit ENTER again to confirm the delete

The GPS will be cleared of all waypoints.

13

Uploading Pre-Load Waypoints to GPS

1. Connect GPS cable to GPS and computer 2. Turn on GPS 3. Open Atlas 4. Click GPS menu item and select Upload Waypoints 5. Browser window opens, navigate to

C:\GM_STS\GPS_FILES\PRELOAD_FILES\ select the name of the quad your trapper will be working in double click the name or click it and hit OK

6. Atlas begins uploading the waypoints to the GPS. When it is complete the message “Upload

successful” appears below the Get Waypoints button on the GPS Management Tab

7. Repeat as necessary. Caution: The Garmin holds only 500 waypoints, so don’t load too many quads. If you load 499 preload waypoints, the trapper can only record 1 waypoint before receiving a message that the unit is full. The trapper can delete a target waypoint to make room for a trap placement waypoint but it is a lot of messing around with the GPS to do so.

14

ATLAS Interface

15

Menus

Along the top of the screen is a menu area. This contains items that activate drop down menus when clicked upon. The file menu item is for dealing with tasks related to opening a previously saved project or saving a current map composition from the program to a project file. This allows the user to create a general map arranged a particular way that can be re used instead of having to re-create that map every time the program is used. This menu will also give you the option to export what you see on the current map with a superimposed legend to an image that can be used or printed by most graphics software. The exit item will close all open processes and windows used by the program.

The next item on this menu bar is the view menu. If there is an active layer in the table of contents, the choices on this menu will be available. The first item allows opening the attribute table associated with the active layer. This will let you see all the information about all the individual items within that layer arranged into a table form. Clicking on the display item will launch a dialog that allows you to manipulate the way a layer is displayed. From there you can adjust things like the fill color, line color, point symbols and much more.

The tools menu contains items that are applied to the map area. The first item, project or transform, launches a tool to change the projection of a shapefile that the user chooses. This can be useful in aligning data from different sources. It offers predefined projection parameters for the common projections used across the STS project area. The import STS CSV file item begins the process of importing a comma delimited text file in the STS format. The process will spatially validate all the trap records to the most appropriate intended locations and assist in visualizing any placement errors.

The GPS menu contains items for dealing with GPS specific tasks. When a user selects this menu the map area will shift to the GPS management tab from whatever it is currently set to. For most users the only item to be concerned with is the get waypoints option. This will find the attached GPS unit, download the trap waypoint information, project it into the appropriate UTM zone and NAD 27 datum, and provide a dialog to save the file within the STS directory structure all in one step. If the user declines to save the file right away they can return to this task by choosing the save CSV item that activates when appropriate. Get tracks is an advanced feature which allows a supervisor to download a tamper proof record of everywhere a given GPS unit has been. The upload waypoints item can be used to preload intended sites onto a unit for navigating to trap sites or for QC inspections during the season. CSV -> GPX converts any STS CSV file into universal GPS Exchange format. These files are the official upload standard and are recognized by many commercial programs like Garmin Map Source©. A final item is included to clear the waypoint list at any time.

16

Selecting the FTP menu will activate the FTP tab like the GPS menu did for its tab. This menu only contains an item to unlock the FTP settings. Before the first use, these will be prompted to be updated and then stored on the system thereafter. If a user changes or a setting requires adjustment, the stored information can be updated at any time by using the unlock settings item.

Lastly, the help menu is where the shortcuts are to launch this file and information about the program release number, author, and technical contact - support information.

17

Toolbars

The next region of the interface contains toolbars with buttons to perform the more frequent functions such as navigation, addition of layers and identity. These toolbars can be detached by hovering the mouse over their handle area, then holding down the right mouse button and dragging them to a new anchor location in the window and releasing the mouse button.

The concepts behind using these buttons in the flow of data processing will be covered elsewhere in more depth. First let’s examine what the buttons do by definition. When the program is running you can also hold the cursor over these buttons to get a popup hint of what they do.

18

Buttons A & B

This program uses groups in the legend to organize sets of similar layers. These buttons will move a group down (A) under overlying groups or up (B) on top of underlying groups. This is useful when you need to see what is underneath something that might otherwise be obstructed from view.

Button one

Use this button to select the zoom tool. This is probably the most commonly used navigation tool. When selected the cursor over the map will look like a magnifying glass and by holding down the left mouse button and dragging a rectangle over a region, the map will zoom to that extent. Conversely, the right mouse button will cause the map to zoom out.

Button C Button two

This button toggles visibility of labels assigned to the active layer off or on depending on their current availability.

This tool has zoom functionality exactly opposite of the zoom in tool described above.

Button D Button three

The identify tool is selected by button D. When clicked it changes the cursor over the map region. The cursor can then be moved over a feature of interest and clicked to bring up a dialog of attributes associated with that one selected feature.

The zoom to full extents tool will zoom to the full extent of all layers in the map.

Button E Button four

Button E is used as another way to launch the process of importing a STS formatted CSV file of waypoint data. The file will be spatially validated on screen, made available for correction and visualization.

The pan tool changes the cursor to a hand over the map area. If the right mouse button is depressed while the map is dragged it will move the map extent up, down, left, or right.

Button F

This button is used to view the associated attribute table of the active layer. In the case of a newly opened CSV layer, it also allows you to edit the attributes for corrections and to access the tool for saving the corrected file.

Button five

This button will walk the map through the previous extents. Each time the map changes its display window extent it is saved onto a stack. This tool will allow you to retrieve the previous extents in order off the top of that stack.

Buttons G & H Button six

These buttons allow you to add (G) layers from a stored location or to remove (H) them from the map.

The zoom to layer extent button will bring the extent of the map to the full area of the active layer when selected.

Button I

This button will clear all groups and their contents from the map at once regardless of whether the project or work is saved. Be careful using it.

19

ATLAS Interface - Tabs

When performing a specialized task, other than mapping, the main map area is switched to a tab with features suited to completing that task. Sometimes the switch will happen automatically when the GPS or FTP menus are selected, but they can also be brought forward selecting the appropriate tab in the region just under the toolbars.

The GPS tab contains a "Get Waypoints" button, a message area and a list area. Everything a user needs to do in order to download waypoints is controlled by a single one touch step of pushing that button. When this happens, the program probes every connection that a GPS unit could be attached to until it finds one. It will then detect the make and model and retrieve all the waypoint information, project and parse to the STS standard format and offer to save the result to a CSV file in the proper downloads folder. There is no configuration required of the user to instantly begin retrieving data in the correct format. It is literally a matter of hooking up the unit and pressing the button. GPS units can also be "hot swapped". This means the program does not need to disconnect or restart when a different GPS is hooked up to it. The GPS only needs to remain connected during transfer operations. Trap data are displayed in the list area in a WYSIWYG format. It will appear exactly as it does in the saved CSV file.

20

The FTP tab is used for uploading GPS information to the main database. Lead workers will not be using this functionality because all CSV files will be sent to the St. Paul office and uploaded to the database from there.

The MAP tab is the default work area used for viewing data and will be discussed below.

The divider that separates the legend from the map is adjustable to the right or left. Mousing over this region will cause the cursor to change. When this happens depress the left mouse button to drag the divider in either direction. This adjustment will not persist between sessions though and the interface resets itself when the program is re-started.

Everything you ever needed to know about the Slow the Spread CSV Format

Absolute column positions represent a character’s numerical position in a row when counting from left to right (i.e. Unique ID begins at position 76). The position of the last character varies because the comment portion of the Unique ID can vary due to record type.

The fields in the above string represent the following information in corresponding order:

Pound: Unused Line Number: The actual row number of the line Landmark: Used for comments UTM Zone: Can be 15,16,17,18 Hemisphere: Always North Easting: Always a 6 digit coordinate Northing: Always a 7 digit coordinate Altitude: The physical elevation in meters Date: month/day/year Time: HH:MM:SS Space: Unused 0: Unused Unique ID: A unique identifying number for a trap + a comment string $: Unused

21

The Unique ID string format

Quad based format:

LITNC0066MG0FA Unique ID Comment String ID composed of 5 character Quad Abbrev + Site Number (9 characters total)

The Comment String Format (placed traps)

123456789M A newly placed trap can be: D – Delta Trap; O – Omitted site

In this case there would be no 11th character unless it is an O (omit), then there would be a reason such as:

H - no place to hang, L - landowner denied access, O - obstacle prohibited access

W - wet, R - rough, V - vegetation too thick, S - safety hazard

An example is below:

123456789OW

The Comment String Format (mid-season & final inspections)

123456789MG001FA

A return inspection trap can be:

M – Mid-season; F – Final

If the above is true, then the 11th column will contain a character in the list below, and possibly followed by a trap catch

D – Damaged (Must Have Catch if True) G – Good (Must Have Catch if True) M – Missing (No Catch Possible) I – Inaccessible (No Catch Possible)

A record becomes a QC inspection when the 15th column is appended with a:

P – Pass; F – Fail

In the above case, a reason must also be provided in the 16th column which can be:

A – Assembled incorrectly C – Outside target circle D – Directions to trap incorrect G – Grid set at wrong spacing I – Trap info not recorded correctly M – Multiple traps set at one site R – Record filled out, no trap set S – Trap set too low to ground T – Delta set where milk carton indicated U – Utms recorded incorrectly W – Inspection incorrectly done X – Trapper did not remove trap from field

22