Embed Size (px)
Citation preview
Where’s the Data?Conducting a First-Year Data Audit
Karen Paulson
Asheville, North Carolina – July 2002
The Data Audit Toolkit
Developed Through a Partnership
> The Policy Center on the First Year of College
• John Gardner, Betsy Barefoot, Randy Swing, Mike Siegel, and Marc Cutright
> NCHEMS
• Karen Paulson and Peter Ewell
Generously Supported by:
> The Pew Charitable Trusts
> The Atlantic Philanthropies
Pilot Institutions
Augustana College (IL) Blue Ridge Community College (VA) Lynchburg College (VA) Northeast State Technical Community College (TN) Ohio University Santa Fe Community College (FL) The University of Texas at El Paso University of Cincinnati University of Minnesota Duluth Washington State University
Data Audit
“The process of identifying data resources
and uses wherever they may be within an
institution and gathering them into a useable
information system.”
NCHEMS Administrative Rationale manual, page 15
Basic Premise I
Most of the Data Needed for Meaningful Analysis Already Reside in Institutional Data Systems
> Admissions
> Registration/Transcripts
> Student Surveys
> Assessment Databases
> Individual “Service Office” Records
Basic Premise II
The Major Task Is:
> To Organize These Data in Useful Ways
> To Make Access and Analysis as Flexible and Straightforward as Possible
The Data Audit Process
Is NOT Itself an End
Is a Means for Understanding and Improving the Use of Data and Information on Campus
Key to Improving the First Year of College
“…we tend to assume that all first-year
programs were implemented as planned
and that the experiences of all students
were uniform.”
In Reality There Are Three First-Year Experiences*
The “Official” One
(Designed and Published Plans)
The “Delivered” One
(Actual Institutional Actions)
The “Experienced” One
(Reality Including Student Choice)
* Adapted from Joan Stark
The Main Question….
How can your institution analytically disentangle
the many elements of the first college year and
provide evidence about the effectiveness, or lack
thereof, of related first-year programs, policies,
and procedures?
Some Preliminary Questions
What Is Unique About the First Year of College at Your Institution?
What Is it Like to Walk in a Student’s Shoes at Your Institution?
How Many Surveys, Tests, or Assessments Does a Student Really Experience During the Year?
Does Any Information from These Data Collections Get Communicated Back to the Students? How?
How Are These First-Year Data Used by Faculty and Administrators?
A Quick Self-Assessment
What Are the Primary Sources of Data About Students, Curricula, and Programs—Specifically Related to the First Year of College—Currently in Place at Your Institution?
Are Any of These Underutilized? Why?
What’s Missing About Which You Would Like Information?
Why a Data Audit?
Identify and Inventory Data Sources
Identify and Inventory Data Needs
Support Assessment of...
> What Happened
> What Mattered
Foster a Culture of Data Use
Primary Data Audit Activities
The Supply Side
Conducting a Campus-Wide Examination of Existing Data Sources
The Demand Side
Determining Which Data Are Most Needed for Evaluation, Assessment, and Decisionmaking
Key Components of a Data Audit
Identifying Sources of Data
Inventorying These Data Sources
Compiling Information About Data Sources
Identifying Gaps in Data Sources
Assessing What Data Users Need and Want to Know
Determining Which Analyses (Existing or Suggested) Have Utility for Stakeholders of Your Institution or Unit
Organizing a Data Audit
Institution-Wide Committee
> IR, Student Affairs, Academic Affairs, Advisors, Orientation Staff, Faculty, Registrar Staff, Students, etc.
Team Members
> People Directly Involved with the Institution
> People Directly Involved with Data
> First-Year Professionals not Normally “Data Connected”
> Leader(s) with Broad Campus Support and Respect
The Data Audit Leadership Team
The Right Attitude
Fresh Perspective
Open Attitude
Collaborative Approach
Not a “Gotcha” Mentality
Visiting Potential Data Sites
Reasons for Physical Visits
> Honors Unit Personnel on Their Turf
> Builds Relationships
> Allows You to Read Reactions
> Allow You to Do Immediate Follow-Up and Collect Artifacts
> Discovers Hidden Databases
Types of Data
Official Data
> Registrar Systems
> Admissions
> Institutional Research Offices
> Assessment Offices
“Unofficial” (Guerrilla) Databases
> Unit Records Kept in Local Computers
> Viewed as “Single” Purpose/Disposable Records
Supply Side Questions
Types of Data?
Who Collects Data and Why?
How Complete Are Data?
Where Do Data Go?
“Walking the Process”
Supply Side Questions to Ask
What Kinds of Records or Data Do You Keep on First Year Students?
How Complete Are Data Collected?
How Are Data Entered?
What Schedules Govern Your Data Collection?
How Are Data Updated?
Finding Existing Data
Follow Student “Footprints”
> Time-Based Investigation
Who Collects Data?
> Structures-Based Investigation
Why Are Data Collected?
> Functions-Based Investigation
Supply Side: Items to Collect
Actual Forms and Questionnaires
Data Element Dictionaries
Data Element Definitions
Database Structures and File Formats
Demand Side Questions
Who Are the Key Constituencies?
What Are the Existing Reports and Requirements?
To Whom Are Data and Information Reported?
What, and When, Are the Decision Cycles?
How Current and Accurate Does the Information Need to Be?
What Are the Gaps in Existing Data?
Demand Side Questions to Ask
To Whom Do You Report Data and Information?
What Information Do You Need or Wish You Had?
What Are the Gaps in Data?
How Are Data Used By Others?
Demand Side: Items to Collect
Representative Reports to Constituents
Copies of External Data Reporting Requirements
Looking for...
Completeness of Data Gathered
Availability of Data
Integrity of Data
Consistency of Data Definitions
Who Coordinates Data Collection?
Who Coordinates Use of Data?
Who Controls Data and Data Processes?
Data Audit Output
Synthesized and Coherent Picture of Existing Data
> Data Element Lists
> Data Definitions
> Data Locations and Locus of Responsibility
> Data Collection Timetables
> Plan for Common Data Collection/Sharing
> Identification of Unmet Data Needs
Design of Recommended Data Structure
Define Core Data Elements
Responsibilities and Data Flows
Core Indicators and Calculational Routines
Data Leverage Points
Problem Identification> Persistence Rate> Violations of Prerequisite Sequencing> Student/Advisor Ratios
Context Setting> Achievement by Gender or Race> Patterns of Student Performance> Entering Student Characteristics
To Inform Discussion> Overall Patterns Rather than Anecdotal or Single
Selling Decisions> Creating Buy-In for Action
Using Evidence to Stimulate and Manage Change
Start with Obvious Discrepancies> Between:
• Perceptions and Reality• Designs and Delivery• Expectations and Results
> Among Different Constituencies and Stakeholders
Avoid “Perfect Data Fallacy”> Recognize There Will Always Be “Errors”> Use Triangulation and Multiple Indicators> Know How Good Is Good Enough
Use Information in “Layers”> Avoid Excessive Complexity> Package Data Around Problems> Disaggregate as Needed in Response to Questions and Opportunities
Use Information to Start Discussions, not “Give Answers”> Develop Multiple Interpretations> Establish “Data Dialogue”
The Data Audit Toolkit(coming fall 2002)
Administrative Rationale
> A non-technical overview and rationale for a data audit.
Technical Manual
> A detailed guide, listing data elements and data, as well as step-by-step plans for a data audit.
Thank You!
Karen Paulson, 303.497.0354, [email protected]
Mike Siegel, 828.877.6009, [email protected]
If you have questions or want to discuss the data audit
and analysis project further, please contact:
