SBL FDW - Qualitative Research

8/6/2019 SBL FDW - Qualitative Research

1/27

DataCollection

and Analysis

Qualitative

Research


2/27

Purpose

Researcher as instrument

Inductive data analysis

Participants meaning

Interpretive inquiry

need for complex, detailed understanding of students experiences

follow up to quantitative research

further insight and explanation into relationships between variables

Why qualitative research?

Characteristics of qualitative research project:

(Creswell, 2007)


3/27

Studio-Based LearningInterview Protocol

1. What were your expectations for the course? Have they been met? Why/why not?

2. Did the course keep you interested and motivated to learn? Why/why not?

3. Did you find the process of completing programming projects helpful to you in learningabout computer programming? Why or why not?

4. Has it been easy in this course to learn about other students programming projects? Ifso, have you found it helpful to learn about other students work? Why or why not?

5. Did you receive helpful feedback from the reviews of your programming projects by

others? If so, was it from your TA or from reviews by other students? How, specifically,did it help you?

6. Did you find the process of writing and sharing reviews of others work helpful to you inlearning computer programming? Why or why not?

7. Did you find the process of giving responses in the lab to others comments on yourstudio presentation helpful to you in learning computer programming? Why or why

not?8. Did you receive helpful feedback during your presentations in the lab? If so, was it

from your TA or from students? How, specifically, did it help you?

9. Did you find the process of participating as an audience member in labs where otherstudents presented their work and responded to comments about it helpful to you inlearning computer programming? Why or why not?

10.Did you provide feedback during other students presentations? If not, why not?


4/27

Studio-Based LearningInterview Protocol

11. Have you discussed course topics with other students or gone to them for help in thiscourse? How about the instructor or TA? How often? When/Why?

12. Are you comfortable giving and receiving feedback on computer programming? Are youconfident about the quality of the feedback you receive as well as the feedback you give?

13. Do you think most students in the course have had similar experiences to yours in workingon programming projects in this course? Do you think that the programming projects andlabs provided a learning experience that is shared by all students in the class?

14. In your view, what impacted your learning more: the lectures, the studio sessions or thejGRASP environment?

15. Has taking this course made you feel more comfortable with computer programming? Withinteracting with others about problems and solutions?

16. Has the course increased or decreased your interest in computer science orcomputer/software/wireless engineering? What parts of the course increased yourinterest? What parts of the course decreased your interest?

17. Will you recommend this course to other students? Why or why not?

18. Did you like working in pairs? Why or why not? If not, do you prefer working alone or in agroup of different size? Please explain..

19. Separate from the issue of working in pairs, Is there anything that you would change aboutthe design of the course? Please explain.

20. What are your plans for taking computer science or computer/software/wirelessengineering courses in the future? Do you plan to major in computer science or

computer/software/wireless engineering?


5/27

TraditionalInterview Protocol

1. What were your expectations for the course? Have they been met? Why/why not?

2. Did the course keep you interested and motivated to learn? Why/why not?

3. Did you find the process of completing programming projects helpful to you in learningabout computer programming? Why or why not?

4. Has it been easy in this course to learn about other students programming projects?If so, have you found it helpful to learn about other students work? Why or why not?

5. Did you discuss programming projects with other students and provide feedback tothem? If so, what form did that feedback take (written, verbal)? If not, why not?

6. Were solutions to programming projects made available? Were these discussed inclass? Did you find the availability and/or discussion helpful to you in learning thecourse topics? Why or why not?

7. Did you receive helpful feedback on your programming projects? If so, from whomdid you get such feedback? How, specifically, did it help you?

8. Have you discussed course topics with other students or gone to them for help in thiscourse? How about the instructor or TA? How often? When? Why?

9. Are you comfortable giving and receiving feedback on computer programming? Areyou confident about the quality of the feedback you receive as well as the feedbackyou give?


6/27

TraditionalInterview Protocol

10. Do you think most students in the course have had similar experiences to yours inworking on programming projects in this course? Do you think that the programming

projects and labs provided a learning experience that is shared by all students in theclass?

11. In your view, what impacted your learning more: the lectures, the studio sessions orthe jGRASP environment?

12. Has taking this course made you feel more comfortable with computer programming?With interacting with others about problems and solutions?

13. Has the course increased or decreased your interest in computer science orcomputer/software/wireless engineering? What parts of the course increased yourinterest? What parts of the course decreased your interest?

14. Will you recommend this course to other students? Why or why not?

15. Is there anything that you would change about the design of the course? Pleaseexplain.

16. What are your plans for taking computer science or computer/software/wirelessengineering courses in the future? Do you plan to major in computer science orcomputer/software/wireless engineering?


7/27

Interview Participants

Semester/Year

Studio-BasedLearning Course

Participants

TraditionalCourse

Participants

Fall 2007 5 5

Fall 2008 10 0

Spring 2008 10 0

Fall 2009 7 8

Spring 2009 0 7

Spring 2010 5 5

Total 37 25


8/27

Coding Process

(Creswell, 2007)


9/27

Data Analysis organize data into file folders,

index cards, or computer files convert files to appropriate text

units for analysis either by handor computer

The data generated by qualitative

methods are voluminous. I have found noway of preparing students for the sheermassive volumes of information withwhich they will find themselvesconfronted when data collection hasended. Sitting down to make sense out ofpages of interviews and whole files offield notes can be overwhelming.

(Patton, 1980)

Coding Process


10/27

Data Analysis get a sense of the whole

database write memos in the margins of

transcripts

Coding Process


11/27

Data Analysis describe in detail develop themes through some

classification system

provide an interpretation in lightof personal views orperspectives from literature

In short, no absolute rules exist except

perhaps this: Do your very best with your

full intellect to fairly represent the dataand communicate what the data revealgiven the purpose of the study.

(Patton, 2002)

independentcoding

whole groupdiscussion

pair coding

whole groupdiscussion

Coding Process


12/27

Data AnalysisIn short, no absolute rules exist except

perhaps this: Do your very best with yourfull intellect to fairly represent the data

and communicate what the data revealgiven the purpose of the study.

(Patton, 2002)

Coding Process


13/27




(Patton, 2002)

Coding ProcessData Analysis


14/27




(Patton, 2002)

Coding ProcessData Analysis


15/27

Data Analysis present the data in a text,

tabular, or figure form develop hypotheses or

propositions that specify therelationship among categories ofinformation

Codebook v. 8

Atlas.ti

Coding Process
http://code%20book%20combined%20v%200.8.docx/http://code%20book%20combined%20v%200.8.docx/


16/27

Studio Based Learning vs. Traditional

Coding Process

Self-Efficacy

Context MotivationLearning

Engagement


17/27

Codebook Structure

Primary Themes

Context Learning Motivation

Self-Efficacy

Engagement

C d b k St t


18/27

Codebook StructureQualifiers

QualifiersA qualifier may be attached to a code as a label and helps determine the interpretation. Any

of the following qualifiers can be attached to learning context, learning, motivation,

engagement, and self-efficacy.

Qualifier:Affective

Response to

explicit statement of expressed

feelings and attitude (only

applies to context and learning)

I really enjoyed it, so..

Qualifier:

Expectations of

explicit statement of

expectation before

experiencing the course

I basically just expected to learn some advanced problem solving techniques.

Qualifier:

Evidence ofquality

explicit statement of judgment

with respect to degree of

excellence or effectiveness

I am confident about the quality of the feedback that I give, Im about 50/50 on

the quality of the feedback I receive.

Qualifier:

Evidence of

learning from

explicit statement of knowledge

gained from course experience

(only applies to context)

Did you find the process of writing and sharing reviews of others work helpful to

you in learning computer programming?

Its interesting because get to learn other approaches and learned from other

approaches.

Positive/

Negative/

Mixed/

Neutral

Only code positive/negative/

mixed where explicit

Positive:

Ok, did you find the process of completing programming projects helpful to you

in learning computer programming?

Yes, all of the projects that we had were targeted at us using a specific new data

structure we just learned, or a new something, so with each project I did learn

efficiently how to use a new something.

Negative:

Shamefully I didnt always pay attention in the lectures, otherwise the lectures

would have impacted me more

Mixed:

Did you find that helpful, to learn about other students work?

Yeah, it provided alternative ideas besides the ideas that were in our own group.

Sometimes they were better, sometimes they were worse.

C d b k St t


19/27

Codebook StructureDefinitions/Examples

Examples

Learning Process The way in which someone learns

Revised Blooms

Taxonomy

A. RememberB. UnderstandC. ApplyD. AnalyzeE. EvaluateF. Create

The cognitive process dimension of the

Revised Blooms Taxonomy (RBT) is

constructed as verbs used to define the

process of learning.

Make the following assumptions when

coding:

1. The categories are not structured as a

cumulative hierarchy, meaning that onecategory does not have to build from the

mastery of another category.

2. Code ALL evident processes of learning

.

A. Retrieving, recognizing, and recalling

relevant knowledge from long-termmemory

B. Constructing meaning from oral, written,and graphic messages through

interpreting, exemplifying, classifying,

summarizing, inferring, comparing, and

explaining

C. Carrying out or using a procedure

through executing, or implementing

D. Breaking material into constituent parts,

determining how the parts relate to one

another and to an overall structure or

purpose through differentiating,

organizing, and attributing

E. Making judgments based on criteria and

standards through checking and

critiquing

F. Putting elements together to form a

coherent or functional whole;

reorganizing elements into a new

pattern or structure through generating,

planning, or producing(Anderson & Krathwohl, 2001, pp. 67-68)

A. NO EXAMPLES FOUND (Possibly not applicable)B. You get another person to give you an idea, so

potentially you can get new and better ideas tocompare yours against.

C. Yes, it was very nice because one of the interestingthings Ive noticed is that for every single problem

that you try to solve with a program, theres any

number of ways to go about solving it. And youknow, some of them are more efficient time wise,some of them are more efficient memory wise, someof them are just not efficient at all, but, theres just

any number of possible solutions.D. Ive had previous experience with except for some of

the algorithms and I guess the inner workings, likeIve used the data structures that were using before.

But Ive never really, you know, it was just like a

black box. You know, I put stuff in and I get stuffout. But now I have a better understanding of, youknow, how those really work and whats going on

behind the scenes.E. The grading of another groups submission ended

up being very important for solidifying the coursematerial. That one aspect, of actually grading, withall of the weight and responsibility of the instructor,you know, as far as trimming the points andWell,

if forces the grader to basically go back over thematerial that the stuff is covering and to solidify it intheir own mind and be sure that they know whatthey are talking about if they are going to give somecritical input on what somebody else has done.

F. Well, for me learning is easier if Im trying to dosomething. If Im trying to solve a problem, its

easier for me to come up with a solution than tohave someone tell me this is the solution, remember

it. Im more likely to remember how to do somethingif Ive done it than if Im just told how to do it.

C d b k St t


20/27

Codebook Structure

Unit of Analysis

Interviewer: In your view, what impacted your

learning more, the lectures, studio sessions, or

jGRASP environment?

Interviewee: I guess the studios. Shamefully I

didnt always pay attention in the lectures,

otherwise the lectures would have impacted me

more, but I think actually listening to peoples

designs in studio and programming with my

partners was the most learning I got.

(Excerpt from studio based learning course interview)

[Qualifier-Evidence of learning from][Global-Positive][Global-Negative][Studio][Context-Course Structure-Lecture][Context-Course Structure-Studio-Presentation][Engagement-Other]

Interviewer: In your view, what impacted your

learning more, the lectures, studio sessions, or

jGRASP environment?

Interviewee: I guess the studios. Shamefully Ididnt always pay attention in the lectures,

otherwise the lectures would have impacted me

more, but I think actually listening to peoples

designs in studio and programming with my

partners was the most learning I got.

(Excerpt from studio based learning course interview)

[Studio]

[Context-Course Structure-Lecture][Global-Negative]

[Engagement-Other]

[Qualifier-Evidence of learning from][Context-Course Structure-Studio-Presentation][Global-Positive]

C d b k St t


21/27

Codebook Structure

Interviewer: Did you receive helpful feedback

from the reviews of your programming projects

by others?

Interviewee: Yeah, um, it was good like wed

talk about okay heres an idea and people

would throw out like have you thought about

doing it like this? Or, how exactly would you

implement it? Or just help us think through

okay like this is how were thinking about it but

is actually practical? So, definitely, the

feedback was really good.

(Excerpts from studio based learning course interview)

[Qualifier-Evidence of quality][Global-Positive][Q06s-Helpful peer reviews]

[Context-Course Structure-Assignment]

[Context-Course Structure-Studio-PeerReview][Context-Interaction-Receiving peerfeedback]

[Learning-Process-Revised BloomsTaxonomy (RBT)-Apply]

[Learning-Process-Revised BloomsTaxonomy (RBT)-Understand]

Multiple Coding

C d /S b d


22/27

Codes/Subcodes

InteractionInteraction my include helping peers learn, providing/receiving/responding to peerfeedback, TA/instructor feedback

Course

Structure

Distinct classroom elements which together compose the structure of a course. Theseelements can include organization (lecture, labs, studio, programming environment)

and activities (assignment). Some of these elements may be present only in a studioor traditional implementation.

CourseDescriptors

Quantifying attributes of a course

InstructorCharacteristics

Attributes of an instructor or teaching assistant

Process The way in which someone learns

Content Subjects or topics covered in the curriculum of the course

Skills The demonstration of proficiency in performing an action

Style Preference for method of interacting with material for learning

Task ValueMotivation determined by the expectations of success and the value of success (i.e.grades, job utility, graduation, usable skills in the real world)

InterestMotivation determined by the inherent interest in an activity and/or knowledge gained

Engagement The degree of participation and attention given toward an activity

Self-Efficacy The feeling of competence to effectively perform a specific task

Context

Learning

Motivation

C di E l 1


23/27

Coding Example 1Studio Based Learning Course

Interviewer: What were your expectations in 2210?

Interviewee: Um, my expectations were pretty much uh continuous learning of the, I guess, basic constructs

used in computer science. You know, different types of objects that we use to represent data. Um and an

introduction to some of the very basic algorithms that we use to deal with those objects.

Interviewer: And have your expectations been met?

Interviewee: Yeah, Id Id say they have. Umyeah.

Interviewer: Okay. Did the course keep you interested and motivated to learn?

Interviewee: Definitely [laughs]

Interviewer: Tell me. Can you explain that more?

Interviewee: Um, the projects. Uh, it it really was the studio section I think that drove me more than

anything. The..most of the stuff that were learning in the classroom Ive had previous experience with

except for some of the algorithms and I guess the inner workings, like Ive used the data structures that

were using before. But Ive never really, you know, it was just like a black box. You know, I put stuff in

and I get stuff out. But now I have a better understanding of, you know, how those really work and whats

going on behind the scenes.

C di E l 2


24/27

Coding Example 2Traditional Course

Interviewer: Okay. Has it been easy in the course to learn about other students assignments and

programming..programming projects?

Interviewee: Um, notprobably not so much because this has been the traditional approach and we

havent really seen or interacted with other students. Weve all been told to do our work individually. Um, I

mean I guess in the little bit Ive spoken with other students um it seems so, but uh I couldnt, probably

couldnt give you a great answer with that.

Interviewer: Did you discuss assignments and programming projects with other students and provide

feedback to them?

Interviewee: Um, on very rare occasions. Typically I I do a lot of my work on my own and I like to

challenge myself to see ifIcan get it..

Interviewer: Okay, thats

Interviewee: um, instead of asking for help. But but then again, I would uh I would also shhhyou

know if we were working in the lab..if I was working in the computer lab and theres some other people in

there too, they would either say something or I would..they would ask something and I would try to bounce

it off them. So that did happen on on occasion.

C di E l 3


25/27

Coding Example 3Studio Based Learning Course

Interviewer: Um, has it been easy in this course to learn about other students group projects?

Interviewee: Uh, yes actually actuallywas the uh the peer review process I I I think worked out fairly

well.

Interviewer: Did you find the process of writing and sharing reviews of others work helpful to you in

learning algorithms?

Interviewee: Yes, maam.

Interviewer: How?

Interviewee: Uh, just reading the the different approaches that some, some other students take is helpful to

have a second set of eyes on any given problem, so seeing where other students may have used a different

strategy helped me to basically learn more strategies to solve the different..a a given problem.

Lessons L ( d)(ing)


26/27

Lessons Learn(ed)(ing)

Qualitative research demands extensive time and

resources

Limit interview questions

Invest in learning qualitative computer softwareprograms before making decisions about codingstructure

Meet frequently

Form pairs or small groups then discuss as a wholegroup

Inter-rater reliability method

I li ti


27/27

Implications

Codebook directly applicable to CPATH II survey

responses

Codebook directly applicable to other computerscience studio based learning courses and traditionalcourses

Codebook indirectly applicable to other courses withinthe field of engineering

Process applicable to developing different codebook

Documents

SBL FDW - Qualitative Research