Upload
xander
View
19
Download
0
Tags:
Embed Size (px)
DESCRIPTION
“Davenport’s Excellence System Assessment Pilot: Measurable Competencies for Improved Teaching and Learning, Student Certification, and Workplace Success” Wynalda TLI Presentation August, 2014 Dr. Irene Bembenista: Vice Provost for Assessment & Graduate Studies - PowerPoint PPT Presentation
Citation preview
“Davenport’s Excellence System Assessment Pilot: Measurable Competencies for Improved Teaching and
Learning, Student Certification, and Workplace Success”
Wynalda TLI PresentationAugust, 2014
Dr. Irene Bembenista: Vice Provost for Assessment & Graduate StudiesDr. Wayne Sneath: Program Director Experiential Learning
Mike Carey: Department Chair, Marketing
Audience Question Cards:
Please include e-mail address& questions for follow up
Session Objectives
To explain the purpose and design of the Excellence System Assessment Pilot
To explain the proposal for certification of students in the Excellence System
To present data on the assessment of all nine Excellence System competencies in courses from two years of the pilot
To generate discussion on the implications of this ongoing assessment on improved teaching and learning and workplace success
The DU Excellence System History
The Mission of Davenport University is “to prepare individuals and organizations to excel in the knowledge-driven environment of the 21st Century.”
To that end, the Davenport University Excellence System was created in partnership with employers who helped to define and reinforce the necessity of these competencies.
The Excellence System consists of nine professional competencies necessary for graduates to engage in life long-learning and succeed in their chosen profession.
The Provost’s Charge
Committee Formed in January, 2012
“The committee shall use the DU Excellence System to develop a methodology and a process that will validate students graduating from DU shall have achieved the skills necessary for each of the DU principles of excellence”
Committee Proposal for Excellence System Certification
Summary:
1. Assessment of proficiency in all 9 Excellence System competencies in key Foundations of Excellence courses & provision of feedback via the development of an electronic portfolio system
2. Assessment of proficiency in all 9 Excellence System outcomes in mid- level and capstone-level courses in each program via the portfolio system
3. Determination of minimum cumulative GPA in all assessed courses
What is Proposed as Certification in the Excellence System?
Minimum score at the Mastery Level in each Excellence System Competency via analysis of assignments from capstone and/or
other program courses (score TBD via pilot)
+
Minimum cumulative GPA in all assessed courses in the Foundations + Program Level (GPA TBD)
= Certification
Year I: Goals of the Pilot
1. Determine a baseline of performance in all 9 Excellence System
competencies by examining required assessment assignments from three Foundations of Excellence courses:
COMM 120: Presentation TechniquesRAA: Final Informative or Persuasive SpeechCompetencies Assessed:• Ethical Reasoning and Action• Information & Tech Proficiency• Professional Communication
ENGL 311: Professional WritingRAA: Group ProjectCompetencies Assessed:• Critical & Creative Thinking• Analysis & Problem Solving• Written Communication• Leadership & Teamwork
SOSC 201: Diversity in SocietyRAA: Experiential Learning PaperCompetencies Assessed:• Global & Intercultural Competence • Civic and Social Responsibility
Year 1: Goals of the Pilot
2. Determine performance in all 9 Excellence System competencies at mid-level and capstone-level courses in programs:
Marketing was the pilot program with assessment in:
MKTG 213: Advertising Foundations MKTG 441: Strategic Marketing (Capstone)
3. Create common rubrics for all 9 Excellence System competencies utilizing cross-college faculty teams
4. Report data to the Provost with recommendations by
May 1, 2013, including a recommendation for development of an electronic portfolio system
through which chosen assignments can be deposited and assessed by course-level faculty
Some Important Questions to be Answered via the Pilot
1. What are the appropriate Foundations of Excellence, Foundations of Business, Technology, or Health, and/or Program courses in which to assess the competencies?
2. Is a review of course RAAs via a common rubric the best means of assessment? Should we recommend nationally normed exams or create our own?
3. How can we effectively train all instructors in these courses to perform consistent assessment of competencies and report this data?
4. What are the best means to assess competencies not easily evaluated by a team via an artifact (technology proficiency, teamwork, professional communication?)
5. What role should ongoing assessment teams play for the Excellence System? Are they cross-college or only by Program? Should they be a “quality check” on course-level instructor assessment scores?
Process for Collection & Assessment of RAAs
• Faculty in chosen sections require students to post RAAs or RAAs are downloaded from existing course shells
• A sample of all RAAs submitted were downloaded & identifying information was removed
• 4 person, cross-college faculty assessment teams were formed and trained in inter-rater reliability and sample scoring; teams read samples to calibrate before scoring independently
• Teams assessed 6 of 9 competencies
• Course faculty assessed “Observable” competencies of Information & Technology Proficiency, Leadership & Teamwork, and Professional Communication
E-Portfolio System Vetting Committee
Purposes:
• Research options for an e-portfolio system which would assist with ongoing Excellence System assessment
• Recommend a system with proposed costs and implementation timeline to the Provost as part of the pilot
• Portfolio system should serve multiple purposes:
ES Assessment & Student Formative Feedback
Faculty Rank & Promotion Dossiers Student Career Development Prior Learning Assessment Portfolios
Year One Pilot Results for:
Global & Intercultural Competence
Irene
Global & Intercultural Competence
Graduates understand that working and succeeding in an inclusive, multicultural, and international world involves complex issues present in diverse environments
1. Evaluates and demonstrates culturally-appropriate behavior in diverse domestic and global contexts
2. Interprets issues and events from diverse domestic and global perspectives
3. Demonstrates an understanding of the dynamics of power, privilege and inequality and appropriate interventions/responses to bias in diverse domestic and global contexts
4. Demonstrates awareness of personal culture and biases and how these influence interactions/relationships in diverse domestic and global contexts
5. Demonstrates an ability to interact successfully with people from diverse domestic and global perspectives
6. Understands and examines differing value systems to determine appropriate courses of action and decision making in the diverse work environment and the domestic and global community
-- Introductory Level—SOSC 201’s Experiential Learning Paper
-- Reinforced Level—MKTG 213’s Group Advertising Plan
-- Mastery Level—MKTG 441’s Group Strategic Marketing Plan
RAAs Mapped to GIC
-- Only 32% of RAAs required a 3rd reader, thus 68% agreement
-- AACU’s standard—reliable at 70% or greater when two readers are within one level of scoring agreement on a rubric
-- GIC was one of the highest reliabilities of the 9 Excellence System competencies assessed in Year One
Inter-Rater Reliability for GIC
-- Mean score was 6.7 of 18 rubric points (6 outcomes x 3)
-- 94% of students were proficient in this competency in SOSC 201
-- 27% of these scored above the Introductory Level
Outcome 1 Outcome 2 Outcome 3 Outcome 4 Outcome 5 Outcome 60.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40 1.281.34
1.05
1.30
0.98
0.78
0.540.58
0.62 0.620.70
0.64
Global & Intercultural Competency Results by Rubric Outcome In SOSC 201
MeanStandard Deviation
Scale 0-3
2012-13 Results: GIC in SOSC 201
-- 94% proficiency at Introductory Level does NOT demonstrate the global aspect of this competency
-- GIC Rubric at Introductory Level assesses either domestic OR global competence
-- GIC Rubric at the Mastery Level assesses both domestic AND global competence
94%
6%
% of Students Proficient in Global & Intercultural Competence at Introductory Level in SOSC
201
Proficient Not Proficient
N=110
Implication I
-- Students scored lowest in GIC Competency Outcome 6:
“Understands and examines differing value systems”
Implication II
Aggregate Results ofAll 9 Excellence System
Competencies
Percentage of Pilot Students Proficient vs. Not-Proficient in all Excellence System Competencies:
Year One Pilot Data
Global & In
tercu
ltural
Competency:
SOSC
201 (N=1
10)
Civic &
Socia
l Resp
onsibilit
y: SO
SC 201 (N
=110)
Ethica
l Reaso
ning & Acti
on: COMM 120 (N
= 60)
Critical
& Cre
ative
Thinkin
g: EN
G: 311 (N
= 160)
Analysis
& Pro
blem Solvi
ng: EN
GL 311 (N
= 160)
Lead
ership &
Team
work: EN
GL 311 (N
=234)
Informati
on &Te
chnology
Proficie
ncy: COMM 120 (N
=227)
Writt
en Communication: E
NGL 311 (N
= 160)
Professi
onal Communica
tion: COMM 120 (N
= 285)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100% 94%
80%
37%
63%
78% 77%81%
49%
91%
6%
20%
63%
37%
22% 23%19%
51%
9%
Excellence System Competency in Foundations of Excellence Courses: COMM 120, SOSC 201, & ENGL 311
% Proficient
% Not Proficient
Total Unique Students Assessed:
N=1076
Mike
Year One Pilot:
Marketing Program Data
Percentage of MKTG441 Students Proficient v. Not Proficient by Excellence System Competency
Critical
& Cre
ative
Thinkin
g N= 2
4
Analysis
& Pro
blem Solvi
ng N= 2
4
Lead
ership &
Team
work N
= 41
Informati
on & Te
chnology
Proficie
ncy N
=41
Writt
en Communication N
= 24
Professi
onal Communica
tion N= 4
10%
10%
20%
30%
40%
50%
60%
70%
80%
90%
29%
13%
44% 46%
63%
22%
71%
87%
56% 54%
37%
78%
% Proficient% Not Proficient
Total Unique Students As-sessed: N=65
Year One Pilot: MKTG Program
Marketing Program Excellence System Certification Analysis for MKTG 441
0%20%40%60%80%
100%
29%
94%
26%
Comparison of Mastery Competency Proficiency, 2.3 or Higher Program GPA, and Combined Mastery Proficiency & Minimum Program GPA
%
N=65
Correlation Analysis ofMKTG Program GPA to RAA Assessment Performance
2 2.5 3 3.5 4 4.50
10
20
30
40
50
60
Series1Linear (Series1)
MKTG Major GPA
Com
pete
ncy
Scor
e To
tal
Irene
Year Two Pilot Planning
&
Results
Conclusion to Year One Pilot
• Average rubric scores compiled and report data was provided to the Provost with recommendations in June, 2013
• Provost and Deans requested a second year of comparative data
• Work began to plan Year Two of the pilot: 2013-14
Planning Year Two of the Excellence System Pilot
• Score a sample of papers from Fall 2013 courses and compare assessment teams’ scores to those already completed by course faculty to determine reliability of instructor scores as a means to do ongoing assessment
• Continue to refine RAA assignments and Excellence System rubrics
• Use data to determine baseline proficiency scores for certification at the Program Level
• Inform curriculum mapping and construction of new RAA assignments
• All PROGRAMS in the colleges charged to identify courses to assess RAAs for all levels of Excellence System proficiency—I, R, and M
2013-14 Pilot: Phase IIPilot Courses: 12 courses, 80+ sections, 1100+ students in
Fall 2013 & Winter 2014
CONTINUED ASSESSMENT OF THE FOLLOWING COURSES:COMM 120: Professional CommunicationENGL 311: Leadership & Teamwork, Written CommunicationSOSC 201: Global & Intercultural Competence, Civic & Social
Responsibility
ADDITIONAL COURSES ADDED FOR ASSESSMENT IN YEAR TWO:HLTH 101: Critical & Creative Thinking, Analysis & Problem SolvingFRSM 100: Critical & Creative Thinking, Analysis & Problem SolvingBUSN 210: Ethical Reasoning & ActionCISP 112: Information & Technology Proficiency
NEW Marketing Program Courses: MKTG 310, 412, 421, 430, 441
Year Two Pilot Results:General
Total Unique Students Assessed:
2014: N=654
Total Unique Students Assessed:
2013: N=1076
Global & In
tercu
ltural
Competence
--I
Civic &
Socia
l Resp
onsibilit
y--I
Ethica
l Reaso
ning & Acti
on (2013=I)
(2014=R
)
Critical
& Cre
ative
Thinkin
g (2013=R
) (2014=I)
Analysis
& Pro
blem Solvi
ng (2013=R
) (2014=I)
Lead
ership &
Team
work--R
Informati
on &Te
chnology
Proficie
ncy*--I
Writt
en Communication--R
Professi
onal Communica
tion--I0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
18.0
20.0
6.7
5.0
3.5
10.3
7.4
9.9
19.0
9.7
13.2
5.44.6 4.9
5.9 6.3
9.911.2 10.3
12.4
2013 Mean Score
2014 Mean Score 2013 v. 2014 Comparison:
Mean Scores by Excellence System Competency
*The number of ITP Rubric Outcomes was reduced from 8 to 5, CSR from 5 to 4, and APS from 6 to 5 in 2014 reflecting different range scores.
I =4-7
2014 I =3-6 2013
I=4-7
2014 R=8-12
2013I=4-7
2014 I=4-72013 R=8-
12
2014 I=4-72013 R=8-
10
R=8-10
2014 I=4-72013 I=6-11
R=10-14
I=4-7
Excellence System Competency Means 2013 v. 2014
Excellence System Proficiency Comparison 2013 v. 2014
Global & In
tercu
ltural
Competence
Civic &
Socia
l Resp
onsibilit
y
Ethica
l Reaso
ning & Acti
on
Critical
& Cre
ative
Thinkin
g
Analysis
& Pro
blem Solvi
ng
Lead
ership &
Team
work
Informati
on &Te
chnology
Proficie
ncy
Writt
en Communication
Professi
onal Communica
tion0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100% 94%
80%
63% 63%
78% 77%81%
49%
91%
6%
20%
37% 37%
22% 23%19%
51%
9%
58%
82%
14%
84% 83%
72%
97%
48%
96%
42%
18%
86%
16% 17%
28%
3%
52%
4%
2013 % Proficient2013 % Not Proficient2014 % Proficient2014 % Not Proficient
2013 v. 2014 Comparison:Percentage of Students Proficient
vs. Not-Proficient by Excellence System Competency
Marketing Program: Year Two Pilot Results
Mike
2014 Marketing Program Courses & Sample Sizes
Total Unique Students Assessed:
2014: N=162
GIC (MKTG 421: N=26)CSR (MKTG 310: N=0)ERA (MKTG 310: N=17)CCT (MKTG 430 N=40)APS (MKTG 412: N=46)LT (MKTG 441: N=33)ITP (MKTG 412: N=46)WC (MKTG 421: N=26)PC (MKTG 441: N=33)
Marketing Program Competency Means 2013 v. 2014
Global & In
tercu
ltural
Competence
--M
Civic &
Socia
l Resp
onsibilit
y*--M
Ethica
l Reaso
ning & Acti
on--M
Critical
& Cre
ative
Thinkin
g--M
Analysis
& Pro
blem Solvi
ng*-M
Lead
ership &
Team
work--M
Informati
on &Te
chnology
Proficie
ncy*M
Writt
en Communication--M
Professi
onal Communica
tion--M
0.0
2.0
4.0
6.0
8.0
10.0
12.0
N/A N/A N/A
8.7 8.8
7.0
11.0
12.0
8.0
5.5
N/A
4.4
7.2
4.9
9.610.0
10.410.0
2013 Mean Score
2014 Mean Score
2013 v. 2014 Comparison:Marketing Student Mean Scores by
Excellence System Competency
*The number of ITP Rubric Outcomes was reduced from 8 to 5, CSR from 5 to 4, and APS from 6 to 5 in 2014 reflect -ing different proficiency range scores.
M =13-18
2014 M =9-12 2013 M=11-15
M=13-18
M=13-18
2014 M=13-18
2013 M=11-15
M=11-15
2014 M=11-152013 M=17-24 M=15-21
M=13-18
Marketing Program Proficiency Comparison 2013 v. 2014
Global & In
tercu
ltural
Competence
Civic &
Socia
l Resp
onsibilit
y
Ethica
l Reaso
ning & Acti
on
Critical
& Cre
ative
Thinkin
g
Analysis
& Pro
blem Solvi
ng
Lead
ership &
Team
work
Informati
on &Te
chnology
Proficie
ncy
Writt
en Communication
Professi
onal Communica
tion0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
N/A N/A N/A
29%
13%
44% 46%
63%
22%
0% 0%
13%
0%
45%39%
15%
30%
100% 100%
87%
100%
55%61%
85%
70%71%
87%
56% 54%
37%
78%
2013 % Proficient2013 % Not Proficient2014 % Proficient2014 % Not Proficient
2013 v. 2014 Comparison:Percentage of Marketing Students Proficient
vs. Not-Proficient by Excellence System Competency
MKTG Program Lessons Learned
Need to Determine the Right:• Course aligned with the Excellence System competency• Assignment (RAA) with the Excellence System competency
Team v. Individual assignments• Rubric language • Training for faculty on scoring using rubrics
Variance in Team v. Faculty scoring in Year Two
Remember:• Not a lot of additional time in reading assignments: most RAAs are read
once for a course grade, but Excellence System rubric scores do NOT impact students’ course grades
• Not all 9 Excellence System competencies are assessed in each course; assessment is spread out across Foundations and Program-Level courses
MKTG Program Next Steps
Davenport University Excellence System Assessment Mapping
Program: Marketing Program 2014-15
Excellence System Competencies INTRODUCED REINFORCED MASTEREDRAA
ASSIGNMENT
Global & Intercultural Competence SOSC201 ECON200 MKTG 421 Country Notebook
Civic & Social Responsibility SOSC201 BUSN225 MKTG 310 Ethics/CSR Paper
Ethical Reasoning & Action ENVS125 BUSN210 MKTG 310 Ethics/CSR Paper
Critical & Creative Thinking ENGL109 LEGL210 MKTG 430 New Product Development Plan
Analysis & Problem Solving ENGL110 STAT220 MKTG 412 Primary Research Project
Leadership & Teamwork ENGL110 ENGL311 MKTG 441 Strategic Marketing Plan
Information & Technology Proficiency COMM120 ACCT202 MKTG 412 Primary Research Project
Written Communication ENGL109 ENGL311 MKTG 421 Country Notebook
Professional Communication COMM120 ENGL311 MKTG 441 Strategic Plan Presentation
Current Work &
Overall Project Next Steps
Wayne
Curriculum Mapping, CRIRs, and ES Data Collection
Summer 2014 Work
1. Excellence System competencies mapped at I,R, and M levels for all academic programs across the colleges
2. CRIR revisions to identity RAAs to be assessed against ES common rubrics
3. Development of an Excellence System assessment website:
www.davenport.edu/academics/assessment/excellence-system
4. Development of Google Drive score sheet system for 9 Excellence System
competencies to be used by course faculty to report data
Project Next Steps
• Determination on final criteria for Excellence System Certification
• Begin assessment in all sections of all identified Excellence Systems assessment courses across all academic programs in Fall 2014
• E-Portfolio pilot with College of Urban Education cohort (Fall 2014)
• Broader portfolio pilot sample (Winter 2015)
Audience Questions and Comments