Assessing Pedagogical Research Breakout Session

Ron Nerio (CUNY), Jessica Joyner (Brooklyn),

Tracy Steffy (KBCC), & Maria Bartolomeo-Maida (KBCC)

January 10, 2017

Some Essentials of Pedagogical Assessment

• 1988: A national movement toward assessment begins

• What makes a good assessment research problem?:• It should be widespread and significant

• It should be solvable

• What makes a good research question?• Measurable

• Control group?

• Outcome can lead to long-term planning

• Can reasonably generate results to share

• Potentially fixable

Essential Strategies for Planning Assessment

• Set expectations: Agree on goals and objectives for learning

• Create a written statement and plan• Set expectations for success• Create a statement describing how students can demonstrate mastery of course or program goals

• Select and design methods and data collection approaches

• Encourage students to provide comments on assessment – they are often closest to the data and can sometimes offer better interpretations

• Carefully plan out timing

• Examine-share-act on findings

Sample Planning Matrix(source: Banta, Trudy W. and Catherine A. Palomba,

Assessment Essentials: Planning Implementing, and Improving Assessment in Higher Education, 2nd. Ed., Jossey-Bass, 2015).

1. What general learning

outcome are your seeking

2.How would you know it if

you saw it?(What will the

student know or be able to do?)

3.How will you help students

learn it (in class or out of class?)

4.How could you measure each of the

desired behaviors listed

in #2?)

5.What are the assessment

findings?

6.What improvements might be based on assessment

findings?

Direct and Indirect Methods

Direct Methods

• Objective exams, essays, oral exams

• Case studies

• Design competitions

• Oral presentations

• Simulations

• Portfolios

• Performance assessment

• Juried activities with outside panels to rate work

Indirect Methods

• Ask students to reflect on what they have learned and experienced• Reflective statements

• Journal writing

• Open-ended comments on surveys

• Performance assessment • Using student activities/products to evaluate

knowledge/skills/

development

• Interviews

• Focus groups

• Surveys

• Self-ratings of learning

Quantitative vs. Qualitative Assessment

• Banta/Palomba: Interest is growing re: qualitative assessment

• Qualitative assessment “provides descriptions of learning in addition to mere numbers on tests or performances” (Banta and Palomba, 2015: p. 81)

• Example: examining materials in students’ portfolios and writing a narrative summary of how students have grown/developed

Aggregating Results in and Across Courses

• Possible If:

• Precise articulation of desired outcomes – and how outcomes can be identified

• Identify criteria that signify levels of performance (outstanding/good/fair, etc.)

• Consistently assign scores for each element of each assignment

Learning objectives:

• Scientific research is a detailed process investigating [everyday] observations and subsequently inspired questions.

• Microbial diversity is vast and only a small proportion can be grown under laboratory conditions.

• There is a microbial community all around us within unique environments that we can benefit from and [positively/negatively] influence.

AREM: Authentic Research Experience in Microbiology

Figure credit: PCSelemliteracy

Bloom’s Taxonomy

What advantage does a metagenomics approach have on a classical culture-dependent assessment of bacterial community diversity? a)A metagenomics approach can accurately identify a greater fraction of the total bacterial diversity in a sample relative to the classical assessments of diversity.

b)A metagenomics approach is safer because it does not require culturing potentially harmful bacteria.

c)A metagenomics approach is more cost effective because it can identify hundreds of different types of bacteria without requiring multiple culturing conditions.

d)All of the above

Approximately what percentage of the total species diversity from an environmental or clinical sample can be cultured under standard laboratory conditions?

a)<10%

b)Between 10-25%

c)Between 45-60%

d)Between 80-95%

e)>90%

What is meant by relative abundance of a bacteria species within a sample?

a)The diversity of species within a given sample

b)The percentage of all the species known to exist in the world that are present in a given sample

c)The representation of each species present in the sample as a percentage of the total bacteria in the sample

d)The abundance of an individual species relative to the global standard for that species

e)None of the above

What is extracted from an environmental sample using the Mo Bio Power Soil or Power Water kit?

a)Bacterial DNA

b)Bacterial RNA

c)Only eukaryotic DNA and RNA

d)Total DNA

What factor(s) might influence the diversity and dynamics of a microbial community in an urban setting?

a)The level of human traffic in the area

b)The presence of food waste

c)The porosity of the surface

d)The presence of trees or plants

e)All of the above

Which pair of samples would you hypothesize to have the most similar microbial communities?

a)Two soil samples 2m apart in a grassy field

b)Water sampled from a pond and a river

c)Intestinal sample and skin sample from the same person

d)Samples from a termite and a housefly intestine

e)None of the above. Bacterial communities are distinct.

Content Questions

What advantage does a metagenomics approach have on a classical culture-dependent assessment of bacterial community diversity? a)A metagenomics approach can accurately identify a greater fraction of the total bacterial diversity in a sample relative to the classical assessments of diversity.

b)A metagenomics approach is safer because it does not require culturing potentially harmful bacteria.

c)A metagenomics approach is more cost effective because it can identify hundreds of different types of bacteria without requiring multiple culturing conditions.

d)All of the above

Approximately what percentage of the total species diversity from an environmental or clinical sample can be cultured under standard laboratory conditions?

a)<10%

b)Between 10-25%

c)Between 45-60%

d)Between 80-95%

e)>90%

What is meant by relative abundance of a bacteria species within a sample?

a)The diversity of species within a given sample

b)The percentage of all the species known to exist in the world that are present in a given sample

c)The representation of each species present in the sample as a percentage of the total bacteria in the sample

d)The abundance of an individual species relative to the global standard for that species

e)None of the above

What is extracted from an environmental sample using the Mo Bio Power Soil or Power Water kit?

a)Bacterial DNA

b)Bacterial RNA

c)Only eukaryotic DNA and RNA

d)Total DNA

What factor(s) might influence the diversity and dynamics of a microbial community in an urban setting?

a)The level of human traffic in the area

b)The presence of food waste

c)The porosity of the surface

d)The presence of trees or plants

e)All of the above

Which pair of samples would you hypothesize to have the most similar microbial communities?

a)Two soil samples 2m apart in a grassy field

b)Water sampled from a pond and a river

c)Intestinal sample and skin sample from the same person

d)Samples from a termite and a housefly intestine

e)None of the above. Bacterial communities are distinct.

Content Questions – w/ Bloom’s Taxonomy

UNDERSTAND

REMEMBER ANALYZE/EVALUATE

APPLY

REMEMBER

UNDERSTAND

Student Responses

0

20

40

60

80

100

120

140

1 2 3 4 5

Procedure

1 2 3 4 5

Influence Microbes

1 2 3 4 5

Similar MicrobiomesQ4 Q5 Q6

Self-reported learning gains1=Little gain, 5=V. large gain

Correct Wrong Blank

Post-surveys Spring 2015

Reading and Writing to Learn: Do Required Writing Assignments Promote Reading

Compliance and Student Engagement in Social Science Classes?

Tracy Steffy & Maria Bartolomeo-Maida

Kingsborough Community College

Components of the Learning Journal:

1. Which two things were the most interesting for you to read about in this chapter and why?

2. How does this chapter relate to you and/or the people in your life in some way? AND Why is it important for you to have knowledge about the content of this chapter?

3. Reflecting critically on this chapter what important issues does it raise that could be/ need to be addressed further?

4. Generate at least one question that came up for you as you were doing this reading? This question should be something not easily answered from doing the reading. It should be thoughtful.

Why is there non-compliance with reading?

• Burchfield & Sappington (2000) reported that only 1/3 of students completed their text assignment on any given day.

• Reasons for not reading include all of the following:- Lack of motivation- Poor understanding of the role of the assigned reading- Students may be too busy or they forget- Assignments may not be structured enough- Students may not comprehend what they are reading and

become frustrated.- Students believe that faculty are going to duplicate the material

in the lecture, anyway.

Findings from first round of data collection-spring 2014

• A slightly modified version of the assignment was piloted in a Developmental Psychology class.

• 89.3% of the 75 students enrolled across two courses made some attempt at the learning journal assignment. Of those who completed the assignment (67 students), 70.1% received full points. However, not all students completed the required number of chapters.

• 13.1% of students reported that they did no reading at all for class.

• When asked why they don’t read or read minimally, 46.3% reported that they do not have the time and 43.9% reported that they get more from classroom lectures.

• Students expressed that: they favored the low stakes writing in comparison to other writing assignments, the assignments allowed them to think critically, apply the material they are learning about, and have another opportunity to express themselves.

Current Research Questions & Hypotheses:

1.Does requiring students to complete graded written assignments about weekly reading assignments promote reading compliance?

• Hypothesis 1: Required, weekly writing assignments based on class readings will promote reading compliance.

2. Do the learning journal assignments encourage comprehension and student engagement with assigned readings?

• Hypothesis 2: Students will demonstrate reading comprehension and student engagement through their journaling.

• Hypothesis 3: Comprehension and engagement with readings will promote class participation.

Data Collection

1. Learning Journals Assignments

• Students journals were evaluated according to a rubric that was utilized in both classes and were awarded points reflecting how effectively the students

fulfilled the assignment

2. Survey of Students

• At the end of the semester, students in the class assigned the learning journals were surveyed about their experience with completing the readings and

journals.

3. Class Participation and Grades

• The Co-PIs observed and assessed levels and quality of class participation. Exam grades, semester grades and individual student grades are being examined.

Fine tuning from Spring 2014 to Fall 2015:

• Created a Grading Rubric

• Had students submit journals each time a new chapter was due (versus mid semester and end of semester)

• Provided sample learning journals to students

• Provided them with a reading about critical thinking

• Modified the questions slightly so that we have less ambiguity and better critical thinking

• We decided to have a control group and more closely examine test scores and participation

• We wanted to see if there was improvement in the quality of journals over time

Instructions for the Assignment:

Each journal entry should be typed and needs to be 1-1.5 pages. You can’t do this assignment adequately if you are writing less than one page. Also, you can not say that you do not have any questions-you must generate a question for #4. Lastly, please bring in two copies of each entry in to class with you.

You will receive 2 points for each journal entry tallying 20% of your grade in total. Your journal entries should be based on the chapter readings from your textbook. They are due the day that the chapter will be discussed. They will not be collected after that time.

Grading Rubric for the Learning Journals:

• Excellent (2 points)- Student addressed all questions with clarity and depth. Student meets or exceeds suggested length of journal entry and raises relevant issues and vital questions. Student shows critical exploration and application of the questions and sound comprehension of the material.

• Good (1.5 points)- Student addressed all questions with clarity. Student meets length of journal with satisfactory level of depth. Student raises relevant issues and shows exploration of the questions. Student can apply the material and comprehends it.

• Fair (1 point)- Student shows exploration of the questions but answers do not meet the requirements for the entry. Student did not answer all questions or there is only a basic demonstration of comprehension and application.

• Unsatisfactory (.5-0 points)- Student answers do not show depth and clarity. Student does not show comprehension and application of the material. Answers seem superficial and student does not raise questions and issues.

Findings from second round of data collection- Fall 2014

66 students completed surveys (POL51 and PSY11)/ N=52 for demographic data

-92%.3% of students reported that they had a textbook for class-76.2% reported that they read 1-5 hours per week. 4.8% are not reading at all. -When asked why they do not read or reading is minimal, 30% said that there are too many details in the book, 20% said they are not finding the time, and 50% indicated multiple items.-When asked if they read for other classes, 18.9% reported “No”-When asked if they had to do learning journal assignments for their other classes, 85.7% said “no”.

Findings from fall 2014- continued

- 42.1% of students agreed and strongly agreed that they had a hard time finding time to complete the journals.

- 74% agreed and strongly agreed that in order to write a learning journal, they read most of the chapter.

- 72% agreed and strongly agreed that the journal assignment fostered critical thinking skills.

- 75% agreed and strongly agreed that the journals gave them another opportunity to express themselves.

- 75% agreed and strongly agreed that they could have put more effort into the learning journal.

Preliminary Findings from 2015-2016 academic year

Psychology:

In PSY 11, 33% of students completed all 10 of the journals, 50% completed 6 or more and the average journal score was 1.5 (out of 2). The survey response rate in the journal classes was 96% and In the non-journal classes, it was 73% and 82% in POL 51. In PSY 11 the average for exam grades was 76 in the journal class and 75 in the non-journal class. The average class grade was 77 in the journal class and 73 in the non-journal class.

Political Science:

In POL 51, only 10% completed all 10 journals, although 61% completed 6 or more. The average journal score was 1.37. The survey response rate in the journal class was 81% and 82% in the non-journal class. In POL 51, the average exam grade for the journal class was 75 and in the non-journal class it was 70. The average class grade was 81 in the journal class and in 71 the non-journal class.

Sample Qualitative Responses from Student Self Report Surveys

What did you like about the learning journal assignment for this class?

-“It helped me understand what we learn in class better and what to look forward to. It allowed me to let my curiosity and opinions out.”

-“Journals help me to study in general and help me improve my writing skills.”

-“They gave me an opportunity to relate the material to my own life.”- “II like that when it came to question 3 about critical thinking, you really had to put your mind into it”

-“I like that we can question the professor in the journal.” -“To be honest, I don’t usually read textbooks if my teacher says to. I never even bought the text for my Art History, Psychology and sociology class. Unless there is a specific assignment attached to the reading. These readings helped me be more prepared for tests”

Sample Qualitative Responses- Continued

What did you dislike about this assignment?

- “On the negative—too much writing, maybe it should be extra credit rather than required, it is time consuming and sometimes difficult to type them.”

- “I didn’t like that we have to do it for all the chapters. Don’t have time to do it sometimes.”

- “I didn’t like that the questions were too structured. They should be more open-ended and…it’s almost impossible to turn one in each week because I work full time and don;t have time to read each week.”

Sample Qualitative Responses- Continued

Any other thoughts about this assignment or about how reading tendencies could be increased?

- “The assignment motivated me to read”

- “It prepared me for my exams and it is such a simple way to boost my grade overall.”

- “I liked the fact that it asked for my opinion.”

- “They felt like preparation for class discussion.”

- “I think that if it weren’t for the journal, I wouldn’t have read all the chapters fully.”

- “Almost one journal a week was hard for me…two chapters can combine into one journal.”

Challenges

• Challenges administratively with the courses. Intention was to collect data in Fall 2015 for Pol Sci and Psych but due to low enrollment the Psychology course did not run.

• Because the study occurred during the Spring for psychology, class sizes were generally smaller.

• The assignment became very demanding for the instructors. A lot of grading and tried to provide timely feedback.

• Writing quality in the introductory courses can be poor and, hence, time consuming.

• Many students did not have the textbook and, hence, still did not complete the assignment.

• Students weren’t finding the time to type and print the journals and so they tried to turn it in late or gave us a handwritten journal. Students were having difficulty providing us with two copies.

• END