www.cirtl.net

Welcome to the CIRTL Network’sTeaching-as-Research Capstone Seminar

SURVEY, DATA COLLECTION, INSTRUMENTS AND QUESTIONNAIRES

Director, EERC

Associate Professor and Fulton C. Noss Faculty Fellow

Department of Industrial Engineering

University of Pittsburgh

Session begins at 3-4 ET/2-3 CT/1-2 MT/12-1 PT

• When you join the room please run the Audio Setup

Wizard: Tools Menu->Audio->Audio Setup Wizard

• While we wait for the session to begin, please

feel free to test your microphone and webcam

• If you are experiencing problems and/or have

questions, please type into the chat window

Dr. Mary Elizabeth Besterfield-Sacre

www.cirtl.net

Ways to Interact during theTAR Working Session

• Turn on/off your microphone:

• Raise your hand if you have a question or comment

• Turn on/off your video:

• Use the chat window to add comments, ask questions, or request help

Survey, Data Collection Instruments,

QuestionnairesMary Besterfield –Sacre

Director, EERCAssociate Professor and Fulton C. Noss Faculty

FellowDepartment of Industrial Engineering

University of Pittsburgh

Once Upon A Time…• I was a grad student…• My research focus– Industrial statistics– Quality assurance

• Finished one NSF grant in healthcare and put on another

• Engineering education– Improving the freshman engineering

curriculum – integrated curriculum– I pouted…

Integration of my skills to solve an engineering education problem

• The effort needed a means to determine if change had occurred in students– Exam questions– Projects– Attitudes and perceptions of engineering– Retention in engineering

• I set up my first evaluation plan…– Objectives, strategies, measures of the

objectives

Thought it would take a weekend…

• First researched the area of student attitudes and students leaving the STEM field– Seymour and Hewitt – Talking About Leaving– Others in the STEM areas– Search for other instruments

• None existed• Got a great survey book - Dillman• Developed an initial survey

– Some closed ended responses based on the research, but two critical open-ended responses• Things that enticed students about engineering• Things that turned students off from engineering

• Piloted the 1st survey on Soph, Jr and Sr

Content Validity

Thought it would take a weekend…

• Initial data analysis– Reliability analysis – Crombach’s alpha– Exploratory factor analysis – some hypotheses yes;

others no– Grounded theory approach to open-ended questions –

created new items• Revised the questionnaire

– Conducted verbal protocols or cognitive interviews• Revised the questionnaire – again• Obtained my first IRB Human Subjects

clearance

Thought it would take a weekend…

• Launched it on Freshman engineering class– “Old” programming

• Launched it on incoming Freshman engineering class– Incoming pre attitudes about engineering– Again, at the end of their Freshman year • “New” programming

Thought it would take a weekend…

• Data analysis– Confirmatory factor analysis– Establishment of weights based on the

factor analysis– Item reliability analysis - again

What did we learn?• Differences between the two cohorts of freshmen

– Besterfield-Sacre, ME, Atman, CJ, and Shuman, LJ. “Engineering Student Attitudes Assessment.” Journal of Engineering Education 87, no 2 (April 1998): 133-141.

• Prediction of freshmen who will leave engineering– Besterfield-Sacre, M.E., C.J. Atman and L.J. Shuman. “Characteristics of

Freshman Engineering Students: Models for Determining Student Attrition and Success in Engineering.” The Journal of Engineering Education 86, no 2 (April 1997): 139-149.

• My first grant– Besterfield-Sacre, M.E., M. Moreno, L.J. Shuman, and C.J. Atman.

“Gender and Ethnicity Differences in Freshmen Engineering Student Attitudes: A Cross-Institutional Study.” Journal of Engineering Education 90, no 4 (October 2001): 477 - 489.

Criterion-related Validity - Concurrent Validity

Criterion-related Validity – Predictive Validity

Criterion-related Validity - Concurrent Validity

Since then…• People still ask for it!– I get about 5-8 schools ask per year

• A group at ASU redid the work a few years back to ‘update’ the instrument

• Construct Validity?– Demonstrates an association between the

survey and the prediction of a theoretical trait

Your questions, issues, challenges?

www.cirtl.net

THANK YOU FOR JOINING TODAY’S SESSION

What did you think of today’s session?

Please take a minute right now to fill out a 1-question survey on today’s experience:

We very much appreciate your feedback.

www.cirtl.net

UPCOMING TAR CAPSTONE SESSION

Working Session: Qualitative and Quantitative Data Management and Analysis

March 17, 20143-4 ET/2-3 CT/1-2 MT/12-1 PT

Facilitated by:Mark Urban-Lurain,

Associate Professor, Center for Engineering Education Research, Michigan State University

To sign up to hear about these and other CIRTL events, email info@cirtl.net.