A Note: Accessing · Web viewWe’ll be using Power Pivot and the Analysis ToolPak during today’s training. In the case that you cannot see Power Pivot in the Customize Ribbons

Day 1 - Introduction to Data Science and Visualization with Excel

Covers: Preparing and Formatting Data; Transforming Data with Formulas; Pivot Tables

Goal: Create a dashboard to view IPEDS completer data by CIP or SOC code and County or LDD

Contents

A Note: Accessing The Reference Files...........................................2

Section 1 – Querying And Loading Data..........................................5

Getting Started – Customizing Excel Options....................................................................................................... 5

Getting Started – Connecting To IPEDS Data........................................................................................................ 6

Getting Started – Connecting To Award Level Lookup Table...............................................................................11

Getting Started – Connecting To CIP To SOC Lookup Table.................................................................................13

Getting Started – Connecting To Title IV Lookup Table.......................................................................................15

Getting Started – Connecting To County To EDD To CBSA Lookup Table.............................................................17

Preparing Data – Merging Lookups Tables With IPEDS Data...............................................................................19

Preparing Data – Preparing IPEDS FIPS Codes For Merge With Geography Crosswalk.........................................26

Section 2 – Analyzing Our Data Using Pivot Tables And Pivot Charts..................................................................................................28

Analyzing Our Data – Pivot Tables With Power Query........................................................................................28

Analyzing Our Data – Pivot Charts With Power Query........................................................................................31

Section 3 – Analyzing Completers By Geography And 2-Digit Soc.. .34

Preparing Data – Merging Geography Data With Our IPEDS Query.....................................................................34

Preparing Data – Creating 2-Digit SOC And Merging With SOC Group Lookup Table...........................................36

Analyzing Data – Creating SOC Group Completers By Geography Pivot Table.....................................................40

Section 4 – Forecasting Data With The Analysis Toolpak................43

1

Analyzing Data – Moving Average To Forecast Completions...............................................................................43

A Note: Accessing the Reference Files1. When you load the reference files to your computer, the files queries will not be active.2. Navigate to the Data tab -> select Queries and Connections -> the Queries and Connections pane

should open.

3. Double-click the selected query you wish to update. The Power Query Editor opens.

4. An error message should be in the yellow box. 5. In the Applied Step pane on the right hand side of the Power Query Editor -> Select Source at the top

of the Applied Steps pane list -> Select Edit Settings in the yellow error bar.a. If the edit settings bar is unavailable, select Source -> Select the “Settings” wheel.

2

6. In the folder selection menu, Select Browse -> Select the source folder or source file for your query -> Ok -> in the Folder window, Ok.

7. To make sure the data has loaded correctly, click Refresh Preview

3

8. To finish connecting the query by selecting Close & Load

9. Repeat these steps for any queries you need to update.

Section 1 – Querying and Loading DataReference “Day 1 IPEDS Dashboard Finished Queries” for an Excel file with a completed version of this section. To view the data, navigate to Data – Queries and Connections.

Getting Started – Customizing Excel Options1. Open Microsoft Excel and open up a new blank workbook.2. Microsoft Excel has a lot of features that aren’t automatically displayed on the menu ribbon. We’ll

need to change the display options so we can use the Power Pivot function.3. Select File -> Options (bottom left of the green menu).

4

4. Under Excel Options select Customize Ribbon -> Under the Main Tabs, make sure Developer, Add-ins, and Power Pivot are selected with a check and click Ok.1

a. We’ll be using Power Pivot and the Analysis ToolPak during today’s training.

5. In the case that you cannot see Power Pivot in the Customize Ribbons section under Options, we’ll need to enable it under the Developer tab.

6. After selecting and checking Developer under Customize Ribbons, navigate to the Developer tab.7. Select COM Add-Ins -> Select and check Microsot Power Pivot for Excel -> Ok.

1 Additional instructions here : https://support.microsoft.com/en-us/office/start-the-power-pivot-add-in-for-excel-a891a66d-36e3-43fc-81e8-fc4798f39ea8?ui=en-us&rs=en-us&ad=us

5

8. Re-complete Step 4 and check and select Power Pivot -> Ok.

Getting Started – Connecting to IPEDS Data1. On the Menu Ribbon navigate to the Data Tab -> Select Get Data -> From File -> From Folder

2. In the Folder selection window, select Browse -> select the “Access Pulls 2017-2019” folder in the “Course Materials” folder -> Select Open -> Ok.

a. We’ll be merging three csv’s with IPEDS completer data from 2017-2019. b. Power Pivot allows us to use data with millions of rows. Data loaded normally into Excel has

a limit of 1,048,576 rows. c. For the purposes of this training, we’ll be connecting to data with less than a million rows.

Excel still has memory usage limits and surpassing it could result in technical issues for some users that would unnecessarily slow down the training (To Maximize Excel’s Memory Performance, be sure to use the 64 bit version of Excel). A set of IPEDS completer data from 2008 to 2019 is available in the Course Materials folder.

d. Power Pivot also updates queries from other workbooks automatically. Meaning that if the structure of the queried workbook is the same, and the data is updated, when you “Refresh” the query in Excel, the data will update too.

6

3. Select Transform Data

4. Click the double down arrows on the “Content” column.

5. In the Combine Files window, select Ok. A data table should populate the window.6. “Column9” contains the CIP Code, which is a text field. Select “Column 9” -> Select the Transform

tab -> Select Data Type -> Text (replacing Decimal Number)a. CIP Code is a text field displayed using numbers. Similar to FIPS codes to denote geography.

7

7. When the Change Column Type window appears, select Replace Current.a. We know the data type conversion worked because “1” became “01.0000”, which is the

format for a 6-digit CIP code.

8. Unfortunately, our csv files from the Access query of IPEDS don’t have titles. Rename each column as the following by double clicking the title and entering the name in quotations:

a. “Column1” to “UnitID”b. “Column2” to “Institution Name”c. “Column3” to “Address”d. “Column4” to “City”e. “Column5” to “State Abbreviation”f. “Column6” to “Zip Code”g. “Column7” to “State + County FIPS”h. “Column8” to “Title IV Flag”i. “Column9” to “CIP Code”j. “Column10” to “Major Number”k. “Column11” to “Award Level”l. “Column12” to “Total Completers”

8

9. Select the “Major Number” column and select Remove Columns on the top left of the Home tab.a. You can use the Applied Steps window on the right-hand side under Query Settings to undo

any action by clicking the “X” next to each step.b. We only needed that column for our query from Access.

10. On the top menu, select the Add Column tab -> Custom Column. Title the new column “Year” and enter the following formula:

if [Source.Name]="Institution Completers 2016-2017.txt" then "2017" else if [Source.Name]="Institution Completers 2017-2018.txt" then "2018" else "2019"

9

11. HINT: Double clicking the “[Source.Name]” field under Available Columns adds it to your formula pre-formatted. 😊 Excel formulas are CASE sensitive.

12. Click OK and our new column will be generated.13. We can now remove the “Source Name” column by navigating back to the Home tab, selecting the

“Source Name” column, and clicking Remove Column.a. We can also right-click the selected column and select Remove.

14. Now our query is ready to load into Excel. In the left-hand side of the menu, select Close & Load -> Close & Load to

15. The data table should now close and the Import Data window should appear. Select Only Create Connection and check Add this data to the Data Model and select Ok.

16. The Queries and Connections window should appear on the right side of Excel. Here you can manage multiple queries and connections to data.

10

Getting Started – Connecting to Award Level Lookup Table1. Navigate to the Data tab on the main Excel menu.2. Select Get Data -> From File -> From Workbook -> Select “IPEDS Award Level Lookup Table” ->

Importa. The Award Level Lookup Table is from the IPEDS documentation2.

3. In the Navigator window, select “Sheet1” -> Click the dropdown arrow next to the Load option -> Load To


2 https://nces.ed.gov/ipeds/use-the-data/download-access-database

11

5. We’ll merge this data with our IPEDS data (Access Pull 2017-2019) later.6. In the Queries and Connections window, right-click “Sheet1” -> Rename -> rename the query “Award

Level” -> and select Rename when the dialogue box appears

Getting Started – Connecting to CIP to SOC Lookup Table1. Navigate to the Data tab on the main Excel menu.2. Select Get Data -> From File -> From Workbook -> Select “CIP to SOC Lookup Table” -> Import

a. The Award Level Lookup Table is from the NCES documentation.3

3 https://nces.ed.gov/ipeds/cipcode/resources.aspx?y=56

12

3. Excel reads our imported CIP Codes as a number instead of text, so we’ll need to transform the data. In the Navigator window, select “CIP-SOC” -> Select Transform Data.

4. Right-click the column heading for “CIP Code” (clicking a row won’t work) -> Change Type -> Text. In the Change Column Type window, select Replace Current. Values that were stored as “1” should now be stored as “01.0000”.

5. Repeat the steps for the column “CIP Family”.

6. Under the Home tab select Close & Load -> Close & Load to

13


8. We’ll merge this data with our IPEDS data (Access Pull 2017-2019) later.

Getting Started – Connecting to Title IV Lookup Table1. Navigate to the Data tab on the main Excel menu.2. Select Get Data -> From File -> From Workbook -> “Title IV Lookup Table” -> Import

a. The Award Level Lookup Table is from the IPEDS documentation4.

4 Variable OPEFLAG - https://nces.ed.gov/ipeds/use-the-data/download-access-database

14

3. In the Navigator window, select “Sheet1” -> Click the dropdown arrow next to the Load option -> Load To


15

5. We’ll merge this data with our IPEDS data (Access Pull 2017-2019) later.6. In the Queries and Connections window, right-click “Sheet1” -> Rename -> rename the query “Title

IV Status” -> and select Rename when the dialogue box appears

Getting Started – Connecting to County to EDD to CBSA Lookup Table1. Navigate to the Data tab on the main Excel menu. 2. Select Get Data -> From File -> From Workbook -> “County to EDD to CBSA Lookup Table” -> Import

a. Geographies were matched using the StatsAmerica County-EDD crosswalk5 and the Census delineation files County-CBSA crosswalk.6

3. In the Navigator window, select “County to LDD and CBSA data” -> Select Transforma. We’ll need to convert the FIPS codes to text.b. Note that the FIPS codes in our IPEDS data were stored as numbers and not text (this is from

the Access query), so we’ll need to fix those data as well to merge our datasets.

5 http://www.statsamerica.org/geography-tools.aspx6 https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html

16

4. Right-click the column heading for “State Code (FIPS” (clicking a row won’t work) -> Change Type ->

Text. In the Change Column Type window, select Replace Current.a. Alternatively, we can Select “State Code (FIPS) -> Select the Transform tab -> Select Data

Type -> Text (replacing Decimal Number)5. Repeat Step 4 for the columns “County Code (FIPS)” and “State + County FIPS”

6. Under the Home tab select Close & Load -> Close & Load to7. The data table should now close and the Import Data window should appear. Select Only Create

Connection and check Add this data to the Data Model and select Ok.

17

Preparing Data – Merging Lookups Tables with IPEDS Data1. In the main Excel Menu ribbon, select the Data tab -> Get Data -> Combine Queries -> Merge

2. In the Merge window, select “Access Pulls 2017-2019” using the top dropdown, and select “Award Level” using the bottom dropdown.

3. In the top data panel, select the “Award Level” column, and in the bottom data panel select the “Codevalue” column. They should highlight as green. Select Ok.

a. The selection should show a match of 735,835 rows from the first table.b. We want to use a Left Outer Join to keep data from IPEDS and only use matching records

from “Award Level”.

18

4. The Power Query Editor should now load. Click the outward facing arrow box in the heading of “Award Level.1”

5. Select Expand -> unselect “Codevalue” and select “Award Level” and select Ok.

19

6. Select the original “Award Level” column heading (with numbers) -> right-click -> Remove.

7. Double click the heading of column “Award Level.1.Award Level” and rename it “Award Level”.8. Select the Home tab -> Close & Load To9. The data table should now close and the Import Data window should appear. Select Only Create


10. Right-click “Merge1” in the Queries and Connections window -> Rename -> Rename it “IPEDS Data + Award Level” -> in the dialogue box -> Rename.

11. We now need to repeat this process for “CIP-SOC” and “Title IV”. We’ll have to further manipulate our IPEDS data to merge it with “County to LDD and CBSA Data”

12. In the main Excel Menu ribbon, select the Data tab -> Get Data -> Combine Queries -> Merge

20

13. In the Merge window, select “IPEDS Data + Award Level” using the top dropdown, and select “CIP-SOC” using the bottom dropdown.

14. In the top data panel, select the “CIP Code” column, and in the bottom data panel select the “CIP Code” column. They should highlight as green. Select Ok.

a. The selection should match 729,644 rows from the first table. This is because not all the CIP codes from IPEDS are in the 2020 CIP format, which is what our CIP-SOC crosswalk uses.7

b. Use a Left Outer Join.

15. The Power Query Editor should now load. Click the outward facing arrow box in the heading of “CIP-SOC”

7 We would need to crosswalk the CIP codes in 2010 format to the 2020 format to have a perfect match: https://nces.ed.gov/ipeds/cipcode/resources.aspx?y=56

21

Select Expand -> unselect “CIP Code” and select “CIP Title”, “CIP Family”, “CIP Family Title”, “SOC Code”, “Primary SOC” and “Primary SOC Title”, unselect Use original column name as prefix and select Ok

16. Select the Home tab -> Close & Load To17. The data table should now close and the Import Data window should appear. Select Only Create


18. Right-click “Merge1” in the Queries and Connections window -> Rename -> Rename it “IPEDS + Award + SOC” -> in the dialogue box -> Rename.

22

a. You’ll notice 735,985 rows loaded, 150 more than in our original data – this is due to some duplication of CIP codes and is something we could sort out if we had more time.

19. Finally, let’s merge Title IV data to our Query. Title IV information helps identify which programs are eligible to receive government-backed student loans from students.


21. In the Merge window, select “IPEDS + Award + SOC” using the top dropdown, and select “Title IV” using the bottom dropdown.

22. In the top data panel, select the “Title IV Flag” column, and in the bottom data panel select the “Codevalue” column. They should highlight as green. Select Ok.

23. The Power Query Editor should now load. Click the outward facing arrow box in the heading of “Title IV”

23

24. Select Expand -> unselect “Codevalue” and select “Title IV Status”, unselect Use original column name as prefix and select Ok.

25. Select the “Title IV Flag” column -> Right-click the heading -> Remove.26. Select the Home tab -> Close & Load To27. The data table should now close and the Import Data window should appear. Select Only Create


24

28. Right-click “Merge1” in the Queries and Connections window -> Rename -> Rename it “IPEDS + Award + SOC +TIV” -> in the dialogue box -> Rename.

Preparing Data – Preparing IPEDS FIPS Codes for Merge with Geography Crosswalk1. Double click the “IPEDS + Award + SOC + TIV” query to open it in the Power Query Editor.2. Select the column “State + County FIPS” -> right-click the column header -> Change Type -> Text

3. Select the Add Column tab -> Custom Column -> Title the column “State + County FIPS Text”, and enter the following formula:

25

if Text.Length([#"State + County FIPS"])=4 then "0"& [#"State + County FIPS"] else [#"State + County FIPS"]

4. Now select the original “State + County FIPS” column -> right-click the header -> Remove.5. Click the Home tab -> Close & Load -> Close & Load

Section 2 – Analyzing our Data Using Pivot Tables and Pivot Charts

26

Reference “Day 1 IPEDS Dashboard Pivot Tables and Pivot Charts” for an Excel file with a completed version of this section.

Analyzing our Data – Pivot Tables with Power Query

1. Let’s pivot to creating pivot tables and pivot charts in Power Query before further analyzing our data.

2. In the Excel menu, select the Power Pivot tab -> Manage.3. In the Power Pivot for Excel window at the bottom, select the “IPEDS Award SOC TIV” tab.

4. Under the Home tab, select the Pivot Table dropdown -> Pivot Table.5. In the Create PivotTable window, select Existing Worksheet -> Ok.

6. In the PivotTable Fields window, select the “IPEDS Award SOC TIV” dropdown -> Drag “State Abbreviation” to Rows, “Year” to Columns, and “Total Completers” to Values.

a. We now have a chart with total completions by year for each state!

27

7. Drag “CIP Family Title” to Filters -> In the “CIP Family Title” dropdown menu, select the down arrow -> Select the “+” sign next to “All” -> Select “Precision Production”. Our table should now filter down to “Precision Production” completers.

8. Drag “Primary SOC Title” to Rows under “State Abbreviation”. Now we can see completions for each SOC Title under the “Precision Production” CIP.

a. We’ll learn how to filter down by SOC Group during groupwork.

28

9. What if we wanted to export this pivot table as a table to one of our coworkers? How can I save this in a datatable format. We’ll use the Design tab.

10. Select the Design Tab -> a. Report Layout -> Show in Tabular Formb. Report Layout -> Repeat All Item Labelsc. Subtotals -> Do not show subtotalsd. Grand Totals -> Off for Rows and Columns

11. We could now copy and paste this table and export it elsewhere!a. If we update the underlying data, and “Refresh” our query by selecting the “Refresh”

icon in the Queries and Connections window and then “Refresh” our Pivot Table by selecting PivotTable Analyze -> Refresh, our data will automatically update to change with its data source (as long as the underlying data structure remains the same).

29

Analyzing our Data – Pivot Charts with Power Query1. We can also use Power Query to create charts!

a. These update with the data when refreshed, same as our Pivot Tables.2. Remove “Primary SOC Title” from Rows

3. Select the dropdown arrow next to “Year” -> Deselect “2017” and “2018”

4. Select the PivotTable Analyze tab -> Pivot Chart. In the Insert Chart window, select Bar -> Clustered Bar.

30

5. By scanning this chart, we can see that Texas (TX) had the most completers of Precision Production programs (according to IPEDS data) in 2019.

a. Using the dropdown menus of our chart, we can customize the data. It will also change the data in our Pivot Table.

b. If we wanted to change our Pivot Chart independently of our Pivot Table, we would need to create a new Pivot Chart/Table with Power Query ( same as we made the first one).

31

6. In the Design and Format tabs we can customize the appearance of our chart.7. In the Design tab -> Select Add Chart Element -> Trendline -> Linear.

8. Now we have a State Average trendline to reference for Completions in the selected CIP Family.

32

Section 3 – Analyzing Completers by Geography and 2-Digit SOCReference “Day 1 IPEDS Dashboard With Geography and SOC Group” for an Excel file with a completed version of this section.

Preparing Data – Merging Geography Data with our IPEDS Query1. Now let’s merge “County to LDD and CBSA Data” with “IPEDS + Award + SOC + TIV”

a. Remember we converted the “State + County FIPS” field in “IPEDs…” into a text field so we can merge the two datasets.


3. In the Merge window, select “IPEDS + Award + SOC + TIV” using the top dropdown, and select “County to LDD and CBSA Data” using the bottom dropdown.

4. In the top data panel, select the “State + County FIPS Text” column, and in the bottom data panel select the “State + County FIPS” column. They should highlight as green. Select Ok.

33

5. Click the double outward arrows in the “County to LDD and CBSA Data” column.6. Select Expand ->

a. Unselect “State Code (FIPS)”, “County Code (FIPS)”, “State + County FIPS”, and “State Abbreviation”

b. Select “County Name”, “State Name”, “Economic Development District”, “Census CBSA”, and “Metropolitan/Micropolitan Statistical Area”.

c. Unselect Use original column name as prefix and select Ok.

7. Select the “State + County FIPS Text” column -> Right-click the heading -> Remove.

34

8. Select the Home tab -> Close & Load To9. The data table should now close and the Import Data window should appear. Select Only Create


10. Right-click “Merge1” in the Queries and Connections window -> Rename -> Rename it “Completions with Geography” -> in the dialogue box -> Rename.

Preparing Data – Creating 2-Digit SOC and Merging with SOC Group Lookup Table1. Double-click the “Completions with Geography” query in the Queries & Connections window to open

the Power Query Editor.2. Select the Add Column tab -> Custom Column3. Title your new column “2-Digit SOC” and enter the following formula:

Text.Start([Primary SOC],2)

35

4. Select the Home tab -> Close & Load -> Close & Load5. Select Get Data -> From File -> From Workbook -> Select “2 Digit SOC to SOC Group Lookup Table” ->

Import

6. Select “Sheet1” -> Transform Data.a. We need to make sure “2 Digit SOC” column is stored as text.

7. In the Power Query Editor look at the Applied Steps window, under Query Settings.8. Notice the Changed Type step. Even though I saved the “2 Digit SOC” column as text, Excel

automatically converted it into a number (Excel does this a lot!). 9. Simply press the “x” next to Changed Type and “2-Digit SOC” will save as text.

36

10. Select the Home tab -> Close & Load -> Close & Load to11. The data table should now close and the Import Data window should appear. Select Only Create


12. Right-click “Sheet1” in the Queries and Connections window -> Rename -> Rename it “2 Digit SOC Title” -> in the dialogue box -> Rename.


37

14. In the Merge window, select “Completions with Geography” using the top dropdown, and select “2 Digit SOC Title” using the bottom dropdown.

15. In the top data panel, select the “2 Digit SOC” column, and in the bottom data panel select the “2 Digit SOC” column. They should highlight as green. Select Ok.

16. Click the double outward arrows in the “2 Digit SOC Title” column.17. Select Expand ->

a. Unselect “2 Digit SOC”b. Select “SOC Group” c. Unselect Use original column name as prefix and select Ok.

38

18. Select the Home tab -> Close & Load -> Close & Load to19. The data table should now close and the Import Data window should appear. Select Only Create


20. Right-click “Merge1” in the Queries and Connections window -> Rename -> Rename it “SOC Completions with Geography” -> in the dialogue box -> Rename.

Analyzing Data – Creating SOC Group Completers by Geography Pivot Table1. Navigate to the Power Pivot tab -> Manage.2. In the Power Pivot for Excel window, select the blue arrow (to view all queries) -> Select “SOC

Completions with Geography”

39

3. Under the Home tab, select the Pivot Table dropdown -> Pivot Table.4. In the Create PivotTable window, select New Worksheet -> Ok.

40

5. Select the “SOC Completions with Geography” query in the PivotTable Fields window.6. Drag “Total Completers” to Values, “Year” to Columns, and “SOC Group” to Rows.

a. The “(blank)” values are from unmatched CIP codes (due to the 2010 to 2020 CIP code transition).

7. Drag “State Name”, “Census CBSA” and “Economic Development District” to Filters.a. I’m going to filter down to the “Regional Planning Commission” in the state of Louisiana.b. Feel free to filter down to your own EDD or CBSA (city).

8. In the “Economic Development District” dropdown menu, select the down arrow -> Select the “+” sign next to “All” -> Select “Regional Planning Commission”. Our table should now filter down to “Regional Planning Commission” completers.

41

Section 4 – Forecasting Data with the Analysis ToolpakReference “Day 1 IPEDS Dashboard Final with Analysis Toolpak” for an Excel file with a completed version of this section.

Analyzing Data – Moving Average to Forecast Completions1. Navigate to the Developer tab -> Excel Add-ins -> Select the Analysis ToolPak –> Ok.

2. We’ll be using completions data for our one SOC Group in our selected geography to creating a moving average that forecasts completions with the Analysis ToolPak.

3. Copy and paste the row of the SOC group with the most completions over the 3 year period into a new sheet. Also copy and paste the row labels into that sheet above the completions.

a. For the Louisiana Regional Planning Commission, that’s “Management Occupations”4. The Analysis ToolPak’s Moving Average tool requires 5 years of data (4 iterations), so we’ll need to

spoof some data for the purposes of this exercise.5. Select the “2017” column -> Insert.

42

6. Title the new column “2016”, and to the right of the “2019” column, title the column “2020”.7. In the value cell below 2016 and 2020, enter the following formula:

=RANDBETWEEN(D2,C2)

8. Select all 5 cells with values -> Copy (ctrl-c) -> Right-click -> Paste Options -> Values 9. Navigate to the Data tab -> Data Analysis -> Moving Average -> Ok.

43

10. In the Input Range, select the 5 cells with dataa. In Interval, select 4b. Set Output Range as the cell to the right of “2020” -> Ok.

11. Your moving average now uses 4 years of data to calculate. a. This is a “forecast” of completers for 2021.b. Delete the “N/A”s next to the forecasted number and enter “2021” above it.

44

Documents

A Note: Accessing · Web viewWe’ll be using Power Pivot and the Analysis ToolPak during today’s training. In the case that you cannot see Power Pivot in the Customize Ribbons