ASSESSMENT OF MORAL REASONING AT TWO FAITH BASED INSTITUTIONS David W. Kale, Ph.D. Director of Assessment, Mount Vernon Nazarene University [email protected]

ASSESSMENT OF MORAL ASSESSMENT OF MORAL REASONING AT TWO REASONING AT TWO

FAITH BASED FAITH BASED INSTITUTIONSINSTITUTIONS

David W. Kale, Ph.D. Director of David W. Kale, Ph.D. Director of Assessment, Mount Vernon Nazarene Assessment, Mount Vernon Nazarene

[email protected]@mvnu.edu

Joel Frederickson, Ph.D. Joel Frederickson, Ph.D. Chair, Psychology Dept., Bethel University Chair, Psychology Dept., Bethel University

[email protected]@bethel.edu

ASSESSMENT OF ASSESSMENT OF MORAL REASONINGMORAL REASONING

BETHEL UNIVERSITYBETHEL UNIVERSITY

Moral Thinking: Using Moral Thinking: Using the DIT-2 as a Measure the DIT-2 as a Measure

of Cognitive of Cognitive DevelopmentDevelopment

What does the DIT-2 What does the DIT-2 measure?measure?

A revised measure of moral A revised measure of moral reasoning based on Kohlberg’s reasoning based on Kohlberg’s theory of moral development. theory of moral development.

Focus is on schemas NOT stages.Focus is on schemas NOT stages. Five scenarios (original DIT had six).Five scenarios (original DIT had six). Strong correlations between DIT-1 Strong correlations between DIT-1

and DIT-2 (and DIT-2 (rr=.79).=.79). Example Dilemma from original DIT: Example Dilemma from original DIT:

Heinz & the drugHeinz & the drug

Schema ScoresSchema Scores

Personal Interest Schema Score: The Personal Interest Schema Score: The proportion of items selected that proportion of items selected that appeal to stage 2 & 3 thinking. Stage appeal to stage 2 & 3 thinking. Stage 2 focuses on the direct advantages to 2 focuses on the direct advantages to the actor and on the fairness of the actor and on the fairness of simple exchanges of favor for favor. simple exchanges of favor for favor. Stage 3 focuses on the good or evil Stage 3 focuses on the good or evil intentions of the parties; concern for intentions of the parties; concern for maintaining friendships and approval.maintaining friendships and approval.


Maintaining Norms Schema Score. Maintaining Norms Schema Score. The proportion of items that appeal The proportion of items that appeal to stage 4 thinking. Focus on to stage 4 thinking. Focus on maintaining existing legal system, maintaining existing legal system, existing roles and organizational existing roles and organizational structure.structure.


Postconventional Schema Score (P Postconventional Schema Score (P score) Focus on organizing society by score) Focus on organizing society by appealing to consensus-producing appealing to consensus-producing procedures (majority vote), insisting procedures (majority vote), insisting on due process, and safeguarding on due process, and safeguarding basic rights. Organizing social basic rights. Organizing social arrangements & relationships in arrangements & relationships in terms of intuitively appealing ideals.terms of intuitively appealing ideals.

ExampleExample

Famine Scenario: Read through this Famine Scenario: Read through this scenario and the accompanying scenario and the accompanying “issues” related to the dilemma.“issues” related to the dilemma.

Place the “issues” into the categories Place the “issues” into the categories they are measuringthey are measuring– Personal InterestsPersonal Interests– Maintaining NormsMaintaining Norms– Postconventional ThinkingPostconventional Thinking– Meaningless/pretentious itemsMeaningless/pretentious items

New Index (N2)New Index (N2)

N2 score is a new index and is considered N2 score is a new index and is considered superior to the P score (post conventional superior to the P score (post conventional reasoning score). reasoning score).

There are two parts to this score: the degree There are two parts to this score: the degree to which respondents support post to which respondents support post conventional responses (P score), plus the conventional responses (P score), plus the degree to which personal interest items degree to which personal interest items receive lower ratings. receive lower ratings.

Essentially, the N2 is a score that reflects the Essentially, the N2 is a score that reflects the degree to which respondents will reject bad degree to which respondents will reject bad arguments (personal interest arguments). arguments (personal interest arguments).

Validity & Reliability of DITValidity & Reliability of DIT

Differentiation of various Differentiation of various age/education groups: 30% to 50% age/education groups: 30% to 50% of the variance in DIT scores is of the variance in DIT scores is attributable to level of education.attributable to level of education.

Education and Moral Education and Moral ThinkingThinking

As education level goes up, degree of As education level goes up, degree of postconventional thinking increases. postconventional thinking increases.

0

10

20

30

40

50

60

JC Fr So Jr Sr MS PD Ph.D


Differentiation of various Differentiation of various age/education groups: 30% to 50% age/education groups: 30% to 50% of the variance in DIT scores is of the variance in DIT scores is attributable to level of education.attributable to level of education.

Longitudinal gains: Reviews of a Longitudinal gains: Reviews of a dozen studies of Freshman to Senior dozen studies of Freshman to Senior college students show effect sizes college students show effect sizes of .80, one of the most dramatic of .80, one of the most dramatic effects of college.effects of college.


Sensitive to moral education Sensitive to moral education interventionsinterventions

Reliability is adequate (Test-retest Reliability is adequate (Test-retest and Cronbach alphas in upper .70s, and Cronbach alphas in upper .70s, lower .80s).lower .80s).

Use of the DIT at Bethel for Use of the DIT at Bethel for AssessmentAssessment

Look at cross-sectional and longitudinal Look at cross-sectional and longitudinal differences from Freshman to Senior year. differences from Freshman to Senior year.

We have also analyzed these Freshman to We have also analyzed these Freshman to Senior changes by department.Senior changes by department.

Example: Business department typically Example: Business department typically had low growth. They made some had low growth. They made some changes to the curriculum (and faculty) changes to the curriculum (and faculty) and saw better growth in moral reasoning. and saw better growth in moral reasoning.


Usually we see typical growth from Usually we see typical growth from Freshman to Senior year (.70-.80 Freshman to Senior year (.70-.80 effect sizes). effect sizes).

A small cohort in our degree A small cohort in our degree completion Business Management completion Business Management program saw similar growth.program saw similar growth.

Just beginning to get longitudinal Just beginning to get longitudinal data for our MBA program.data for our MBA program.


Colleague uses pre/post DIT in his Colleague uses pre/post DIT in his course “Being Just in an Unjust course “Being Just in an Unjust World” (essentially a course in Moral World” (essentially a course in Moral Thinking). Typically sees a big Thinking). Typically sees a big change for a semester (about .70 change for a semester (about .70 effect size).effect size).

This is the amount of change you This is the amount of change you would typically see in 4 years!would typically see in 4 years!

Pre/post DIT N2 Scores by Type Pre/post DIT N2 Scores by Type of Courseof Course

0

10

20

30

40

50

60

Pre Post

Being Just course

Comparison courses


A small subset of students who had A small subset of students who had taken this course (N=23) took the taken this course (N=23) took the DIT again four months later.DIT again four months later.

Students not only maintained Students not only maintained growth, but showed a marginally growth, but showed a marginally significant increase. significant increase.

ASSESSMENT OF MORAL ASSESSMENT OF MORAL REASONING AT MOUNT REASONING AT MOUNT

VERNON NAZARENE VERNON NAZARENE UNIVERSITYUNIVERSITY

COMBINING THE COMBINING THE QUANTITATIVE WITH THE QUANTITATIVE WITH THE

QUALITATIVEQUALITATIVE

QUALITATIVE ASSESSMENTQUALITATIVE ASSESSMENT

The Potter BoxThe Potter Box

Definition of situation

Values

Loyalties

Principles

Ralph B. Potter, “The Logic of Moral Argument” in Towards a Discipline of SocialEthics, ed. Paul Deats (Boston Univ. Press, 1972).

DEFINITION OF THE SITUATIONDEFINITION OF THE SITUATION

The letter is the engineer’s personal The letter is the engineer’s personal and professional correspondence.and professional correspondence.

The letter is privileged The letter is privileged communication between the communication between the engineer and his attorney.engineer and his attorney.

What other aspects of the situation What other aspects of the situation would you consider important to would you consider important to consider?consider?

VALUESVALUES

PrivacyPrivacy TruthTruth What other values do you see?What other values do you see? What is your highest value?What is your highest value?

PRINCIPLESPRINCIPLES

Always tell the truthAlways tell the truth

PRINCIPLESPRINCIPLES

Tell the truthTell the truth Protect the privacy of othersProtect the privacy of others

LOYALTIESLOYALTIES

To whom is moral duty owed?To whom is moral duty owed?

- The 17 year old- The 17 year old

- The engineer- The engineer

- Who else has a stake in this - Who else has a stake in this situation that needs to be protected?situation that needs to be protected?

To whom do you have the highest To whom do you have the highest loyalty?loyalty?

MORAL AND ETHICAL MORAL AND ETHICAL THEORIESTHEORIES

Aristotle’s Golden MeanAristotle’s Golden Mean Biblical concept of loveBiblical concept of love UtilitarianismUtilitarianism Rawls’ Theory of JusticeRawls’ Theory of Justice Kant’s Categorical ImperativeKant’s Categorical Imperative

BUILDING A RUBRIC FOR BUILDING A RUBRIC FOR STUDENT SELF ASSESSMENTSTUDENT SELF ASSESSMENT

After dealing with several case After dealing with several case studies, I ask students to answer two studies, I ask students to answer two questions anonymously.questions anonymously.

1. In what ways have you improved 1. In what ways have you improved in your moral and ethical reasoning in your moral and ethical reasoning ability?ability?

2. In what ways would you like to 2. In what ways would you like to continue to improve?continue to improve?


I identified eight dimensions of moral I identified eight dimensions of moral reasoning in their answers.reasoning in their answers.

1. Using clearly worded arguments when I 1. Using clearly worded arguments when I state my position;state my position;

2. Making sure I have good support for my 2. Making sure I have good support for my position (facts, reasoning, statistics)position (facts, reasoning, statistics)

3. Taking the views of others into 3. Taking the views of others into consideration when I make my decision.consideration when I make my decision.

4. Supporting my arguments with biblical 4. Supporting my arguments with biblical truth.truth.


5. Having a clear process for moving 5. Having a clear process for moving all the way to a decision;all the way to a decision;

6. Making my decisions based on 6. Making my decisions based on reason rather than letting my reason rather than letting my emotions get the upper hand;emotions get the upper hand;

7. Taking my time rather than 7. Taking my time rather than rushing into a decision.rushing into a decision.

8. Staying well informed on the 8. Staying well informed on the issues of the day.issues of the day.


I then have students vote on the top five.I then have students vote on the top five. 1. Using clearly worded arguments when I 1. Using clearly worded arguments when I

state my position.state my position. 2. Making sure I have good support for my 2. Making sure I have good support for my

position (facts, reasoning, statistics).position (facts, reasoning, statistics). 3. Taking the views of others into 3. Taking the views of others into

consideration.consideration. 4. Supporting my position with biblical truth.4. Supporting my position with biblical truth. 5. Taking my time rather than rushing into a 5. Taking my time rather than rushing into a

decision.decision.


Building a rubric in this fashion is an Building a rubric in this fashion is an assessment strategy in itself.assessment strategy in itself.

If there is an aspect of moral If there is an aspect of moral reasoning that does not turn up in reasoning that does not turn up in their list that I think is important, their list that I think is important, that tells me either that they have that tells me either that they have not learned it or do not think it is not learned it or do not think it is nearly as important as I do.nearly as important as I do.

QUANTITATIVE ASSESSMENT QUANTITATIVE ASSESSMENT USING THE DIT-2 AT MVNUUSING THE DIT-2 AT MVNU

On the traditional side, we administer On the traditional side, we administer the DIT-2 to our incoming first year the DIT-2 to our incoming first year students, to our Juniors on Testing students, to our Juniors on Testing Day and to our seniors at the end of Day and to our seniors at the end of the Senior Colloquium course.the Senior Colloquium course.

We also administer the DIT-2 to our We also administer the DIT-2 to our nontraditional students at the nontraditional students at the beginning and at the end of their beginning and at the end of their degree completion programs. degree completion programs.

N2 scoresN2 scores

Trad. JuniorsTrad. Juniors

26.5126.51Juniors nat’l. Juniors nat’l. AvAv

32.6532.65

p = <.01p = <.01

Trad. SeniorsTrad. Seniors

35.2835.28Seniors nat’l. Seniors nat’l. AvAv

36.85 36.85

p = .079p = .079


26.5126.51Trad. SeniorsTrad. Seniors

35.2835.28 p = <.001p = <.001


26.51 26.51 Non-trad. Non-trad. JuniorJunior

23.0623.06

p = .148p = .148

Trad. SeniorsTrad. Seniors

35.2835.28Non. Trad. Non. Trad. Sen.Sen.

26.5526.55

p = .0002p = .0002

Trad. And Non-Trad. StudentsTrad. And Non-Trad. StudentsPost conventional Reason. Post conventional Reason.

ScoresScores

0

5

10

15

20

25

30

35

40

Junior Senior

TradNon Trad

SUMMARYSUMMARY

There appears to be strong evidence There appears to be strong evidence from the use of the Defining Issues from the use of the Defining Issues Test at two faith based institutions Test at two faith based institutions that a senior level Christian ethics that a senior level Christian ethics course significantly improves course significantly improves students’ moral reasoning ability.students’ moral reasoning ability.

Using both quantitative and Using both quantitative and qualitiative measures provides richer qualitiative measures provides richer data as to exactly what aspects of data as to exactly what aspects of students’ reasoning has improved. students’ reasoning has improved.

Documents

ASSESSMENT OF MORAL REASONING AT TWO FAITH BASED INSTITUTIONS David W. Kale, Ph.D. Director of Assessment, Mount Vernon Nazarene University [email protected]