Wheres the Data? Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July...

Where’s the Data?Conducting a First-Year Data Audit

Karen Paulson

Asheville, North Carolina – July 2002

The Data Audit Toolkit

Developed Through a Partnership

> The Policy Center on the First Year of College

• John Gardner, Betsy Barefoot, Randy Swing, Mike Siegel, and Marc Cutright

> NCHEMS

• Karen Paulson and Peter Ewell

Generously Supported by:

> The Pew Charitable Trusts

> The Atlantic Philanthropies

Pilot Institutions

Augustana College (IL) Blue Ridge Community College (VA) Lynchburg College (VA) Northeast State Technical Community College (TN) Ohio University Santa Fe Community College (FL) The University of Texas at El Paso University of Cincinnati University of Minnesota Duluth Washington State University

Data Audit

“The process of identifying data resources

and uses wherever they may be within an

institution and gathering them into a useable

information system.”

NCHEMS Administrative Rationale manual, page 15

Basic Premise I

Most of the Data Needed for Meaningful Analysis Already Reside in Institutional Data Systems

> Admissions

> Registration/Transcripts

> Student Surveys

> Assessment Databases

> Individual “Service Office” Records

Basic Premise II

The Major Task Is:

> To Organize These Data in Useful Ways

> To Make Access and Analysis as Flexible and Straightforward as Possible

The Data Audit Process

Is NOT Itself an End

Is a Means for Understanding and Improving the Use of Data and Information on Campus

Key to Improving the First Year of College

“…we tend to assume that all first-year

programs were implemented as planned

and that the experiences of all students

were uniform.”

In Reality There Are Three First-Year Experiences*

The “Official” One

(Designed and Published Plans)

The “Delivered” One

(Actual Institutional Actions)

The “Experienced” One

(Reality Including Student Choice)

* Adapted from Joan Stark

The Main Question….

How can your institution analytically disentangle

the many elements of the first college year and

provide evidence about the effectiveness, or lack

thereof, of related first-year programs, policies,

and procedures?

Some Preliminary Questions

What Is Unique About the First Year of College at Your Institution?

What Is it Like to Walk in a Student’s Shoes at Your Institution?

How Many Surveys, Tests, or Assessments Does a Student Really Experience During the Year?

Does Any Information from These Data Collections Get Communicated Back to the Students? How?

How Are These First-Year Data Used by Faculty and Administrators?

A Quick Self-Assessment

What Are the Primary Sources of Data About Students, Curricula, and Programs—Specifically Related to the First Year of College—Currently in Place at Your Institution?

Are Any of These Underutilized? Why?

What’s Missing About Which You Would Like Information?

Why a Data Audit?

Identify and Inventory Data Sources

Identify and Inventory Data Needs

Support Assessment of...

> What Happened

> What Mattered

Foster a Culture of Data Use

Primary Data Audit Activities

The Supply Side

Conducting a Campus-Wide Examination of Existing Data Sources

The Demand Side

Determining Which Data Are Most Needed for Evaluation, Assessment, and Decisionmaking

Key Components of a Data Audit

Identifying Sources of Data

Inventorying These Data Sources

Compiling Information About Data Sources

Identifying Gaps in Data Sources

Assessing What Data Users Need and Want to Know

Determining Which Analyses (Existing or Suggested) Have Utility for Stakeholders of Your Institution or Unit

Organizing a Data Audit

Institution-Wide Committee

> IR, Student Affairs, Academic Affairs, Advisors, Orientation Staff, Faculty, Registrar Staff, Students, etc.

Team Members

> People Directly Involved with the Institution

> People Directly Involved with Data

> First-Year Professionals not Normally “Data Connected”

> Leader(s) with Broad Campus Support and Respect

The Data Audit Leadership Team

The Right Attitude

Fresh Perspective

Open Attitude

Collaborative Approach

Not a “Gotcha” Mentality

Visiting Potential Data Sites

Reasons for Physical Visits

> Honors Unit Personnel on Their Turf

> Builds Relationships

> Allows You to Read Reactions

> Allow You to Do Immediate Follow-Up and Collect Artifacts

> Discovers Hidden Databases

Types of Data

Official Data

> Registrar Systems

> Admissions

> Institutional Research Offices

> Assessment Offices

“Unofficial” (Guerrilla) Databases

> Unit Records Kept in Local Computers

> Viewed as “Single” Purpose/Disposable Records

Supply Side Questions

Types of Data?

Who Collects Data and Why?

How Complete Are Data?

Where Do Data Go?

“Walking the Process”

Supply Side Questions to Ask

What Kinds of Records or Data Do You Keep on First Year Students?

How Complete Are Data Collected?

How Are Data Entered?

What Schedules Govern Your Data Collection?

How Are Data Updated?

Finding Existing Data

Follow Student “Footprints”

> Time-Based Investigation

Who Collects Data?

> Structures-Based Investigation

Why Are Data Collected?

> Functions-Based Investigation

Supply Side: Items to Collect

Actual Forms and Questionnaires

Data Element Dictionaries

Data Element Definitions

Database Structures and File Formats

Demand Side Questions

Who Are the Key Constituencies?

What Are the Existing Reports and Requirements?

To Whom Are Data and Information Reported?

What, and When, Are the Decision Cycles?

How Current and Accurate Does the Information Need to Be?

What Are the Gaps in Existing Data?

Demand Side Questions to Ask

To Whom Do You Report Data and Information?

What Information Do You Need or Wish You Had?

What Are the Gaps in Data?

How Are Data Used By Others?

Demand Side: Items to Collect

Representative Reports to Constituents

Copies of External Data Reporting Requirements

Looking for...

Completeness of Data Gathered

Availability of Data

Integrity of Data

Consistency of Data Definitions

Who Coordinates Data Collection?

Who Coordinates Use of Data?

Who Controls Data and Data Processes?

Data Audit Output

Synthesized and Coherent Picture of Existing Data

> Data Element Lists

> Data Definitions

> Data Locations and Locus of Responsibility

> Data Collection Timetables

> Plan for Common Data Collection/Sharing

> Identification of Unmet Data Needs

Design of Recommended Data Structure

Define Core Data Elements

Responsibilities and Data Flows

Core Indicators and Calculational Routines

Data Leverage Points

Problem Identification> Persistence Rate> Violations of Prerequisite Sequencing> Student/Advisor Ratios

Context Setting> Achievement by Gender or Race> Patterns of Student Performance> Entering Student Characteristics

To Inform Discussion> Overall Patterns Rather than Anecdotal or Single

Selling Decisions> Creating Buy-In for Action

Using Evidence to Stimulate and Manage Change

Start with Obvious Discrepancies> Between:

• Perceptions and Reality• Designs and Delivery• Expectations and Results

> Among Different Constituencies and Stakeholders

Avoid “Perfect Data Fallacy”> Recognize There Will Always Be “Errors”> Use Triangulation and Multiple Indicators> Know How Good Is Good Enough

Use Information in “Layers”> Avoid Excessive Complexity> Package Data Around Problems> Disaggregate as Needed in Response to Questions and Opportunities

Use Information to Start Discussions, not “Give Answers”> Develop Multiple Interpretations> Establish “Data Dialogue”

The Data Audit Toolkit(coming fall 2002)

Administrative Rationale

> A non-technical overview and rationale for a data audit.

Technical Manual

> A detailed guide, listing data elements and data, as well as step-by-step plans for a data audit.

Thank You!

Karen Paulson, 303.497.0354, Karen@nchems.org

Mike Siegel, 828.877.6009, siegelmj@brevard.edu

If you have questions or want to discuss the data audit

and analysis project further, please contact: