Using Portfolio Assessment to Discover Student
LearningBarbara J. D’Angelo, Ph.D.Arizona State University
Association for Business Communication Annual Convention
November 2009
Background: Written Communication for Managers Junior-level business writing course
Multi-section course 7-10 sections per semester + summer On-campus, hybrid, and online
Redesigned Summer 2007 Industry focus groups Business and writing faculty input Role playing scenario: students create a company Standardized syllabus and assignments
Portfolio for assessment End-of-semester self-evaluation of learning Scored by 1 writing faculty and 1 business faculty
Barbara J. D'Angelo, ABC Convention November 2009 2
Research QuestionsWhich assignments do students choose to
include in their portfolios?
How do students organize their portfolios as an argument to demonstrate their learning?
What persuasive techniques do students use to support their argument for learning?
What tone and style do students use?
Do students present an argument for their learning or to enhance their grade?
Barbara J. D'Angelo, ABC Convention November 2009 3
MethodsRandom sampling
3 portfolios from each section Fall 2008 and Spring 2009 34 portfolios analyzed
Simple counting of assignments
Content analysis of narrative statements Categories derived from
Bower (2003): rhetorical appeals and argumentationScott (2005): form, tone, and appeal
Post-semester feedback from instructors
See proceedings for details (or talk to me later or email me)
Barbara J. D'Angelo, ABC Convention November 2009 4
Assignments Included in Portfolios
Barbara J. D'Angelo, ABC Convention November 2009 5
Fall 2008 Spring 2009
Memo 11 14
Manager Roundtable
10 12
Proposal 10 10
“Branding” 9 11
Letter 8 14
Policy 9 10
Audience Analysis 6 1
Agenda 4 7
Email 4 5
Peer Review 4 5
Research Summary 3 3
Minutes 2 7
AssignmentsAssignment inclusion not related to grading scale
Inclusion seems based on Substantiality of writing or traditional genres Instructor emphasis: instructor feedback indicates this
as high
De-emphasis of meeting documents Lack of understanding or emphasis Increased emphasis during spring = higher inclusion =
awareness of importance?
Process documents Reluctance to show drafts or other “in progress” work?
6Barbara J. D'Angelo, ABC Convention November 2009
Portfolio Organization
26 organized generically Introduction: reflection about courseBody: described artifacts, claims and evidenceConclusion: summed up
7 organized contextuallyBody contained more context
Best vs. worstStrengths vs. weaknessesCourse objectives
Had no influence on raters’ scores
Barbara J. D'Angelo, ABC Convention November 2009 7
Persuasive techniques# of
Portfolios# of
Statements
Ethos 14 19
Logos 23 51
Pathos 17 29
Nomos 21 31
Skills Supported 23 64
Skills Unsupported 18 39
Barbara J. D'Angelo, ABC Convention November 2009 8
Use of PersuasionNomos and pathos not linked to “schmooze”
Not directly linked to a claim or evidence
More expression of student “feelings”
Awareness of audience and course values
Majority (23) of students supported claims 23 used logos to create claim with evidence to support it
13 supported all claimsUnlike findings with FYC students
Indication of more sophisticated writing from upper level students?
8 did not support any claims
9Barbara J. D'Angelo, ABC Convention November 2009
Tone and Style 24 students used positive tone
Excitement about role playing (pathos)
3 students used negative toneDis-satisfied with grade or with instructor?
7 students used intermediate toneNeutral descriptive tone
All students changed from “business” style to “student” style when completing the portfolio
Barbara J. D'Angelo, ABC Convention November 2009 10
Learning or enhance grade?
Learning 28 students made statements related to their growth Only 1 student directly referred to their grade Use of logos and support for claims indicates argument
was for learning
Enhance grade 19 portfolios potentially included “schmooze”
2 flattered course or instructorRest thanked instructor
2 students appealed directly to instructor (using “you”)
Result differs from FYC studies
Barbara J. D'Angelo, ABC Convention November 2009 11
Use for course improvement More emphasis on meeting documents
Importance to our employer stakeholders
Additional requirement to write agendas added for each Manager Roundtable
Requirement to evaluate agendas
Change MRT topics to more completely support written assignments Instructors believed not enough time to go over concepts
needed for assignments
Strengthen research summary to emphasize tie between finding information-analysis-use for persuasion
12Barbara J. D'Angelo, ABC Convention November 2009
FuturePotential change: portfolio scenario to shareholder
or other external audience Maintain emphasis on business style
Continue to de-emphasize mechanicsBut help needed for ESL students
Continue assessment to aggregate additional data
Confirmation of findings related to use of logos/evidence
Add industry evaluators to assessment to include external stakeholders
13Barbara J. D'Angelo, ABC Convention November 2009
Questions?Presentation available at:
http://www.public.asu.edu/~bdangelo/presentations/abc09.ppt
Course syllabus and assignments available at:
http://techcomm.asu.edu/curriculum/twc347
Email: [email protected]
Barbara J. D'Angelo, ABC Convention November 2009 14