HeightWeightI.Q.IncomeBody mass indexAgeTestosteroneHighest grade#years collegeAmount of hair

Measures ofSelf-esteemAcceptance of OthersNarcissismTraditional sex RolesSatisfaction withlife

Qualitative measures with words Defining what makes a difference

Qualitatively and quantitativelyObserve behaviorObserve environLook at documents(diploma etc)Observe appearanceInterview spousePerson, familyKey informantsSurvey friendsGo thru trashCount Big MacWrappers in car

Look, listen andDescribe, describe describeWrite, write, writeAnalyze, analyze, analyze

• QUALITATIVE MEASURING BY USING WORDS (RIGHT HAND COLUMN) REQUIRES DIFFERENT SKILLS AND METHODS THAN USING NUMBERS

Process Evaluations and qualitative researchQuantitative answers Is there a difference? A change, effective? And How much? Are there similarites? Relationships? (correlation) And How much? Qualitative describes! Answers What is it? Why is it? How does it work? And What does it mean?

Qualitative approaches are typically used with process evaluations. . Quality assurance, CQI, TQM, program monitoring, needs assessments, formative evaluations, implementation evaluation and to some degree consumer satisfaction are all formative.

Qualitative research typically does not use measurement, numbers and statistics,. It uses words, descriptions, artifacts and observations!

PROCESS EVAL = QUALITATIVE = IMPROVEMENT OF PROGRAMOUTCOME EVAL = QUANTITATIVE = JUDGMENT ABOUT PROGRAM WORTH

Here are the things that a qualitative researcher does:• observes events and activities as they occur in the program setting.• gets directly involved with the people in the program and sometimes personally

experiences the processes used in the program.• acquires an insiders’ point of view while maintaining an analytic perspective• uses a variety of observational and interviewing techniques and social skills in

a flexible –often not pre-determined way-which allows the evaluator to follow the answers to the research questions wherever they might go.

• produces data in the form of extensive written notes, maps, diagrams and sometimes pictures in order to provide very detailed descriptions.

• watches, listens and collects data• uses artifacts of the program like records, treatment plans, staffing patterns,

absentee rates, dropout rates, ashtray use, lounge use, patterns of wear on equipment, but only to reach conclusions not to count!

• uses observations and/or several forms of interviewing.• reviews the collected data (in the form of written notes, videotapes, in-depth

written interviews, audiotapes.• in reviewing the data, begins to look for repetitive ideas, themes, common

conceptions, incongruities, strengths and weaknesses.• reviews the data again and begins to make certain hypotheses grounded solely

in the collected data!• begins to draw meanings and conclusions from the data collected.• writes a narrative about those meanings and conclusions.

• ALTHOUGH CLEAR OBJECTIVES SHOULD BE A PART OF ANY PROGRAM, THEY ARE NOT AS NECESSARY TO DO A PROCESS EVALUATION.

• HOWEVER YOU STILL NEED TO HAVE SOME SENSE OF A. WHAT YOU ARE MEASURING AND B. HOW YOU WOULD MEASURE IT IN A PROCESS EVALUATION.

• REALLY WHAT WE ARE TALKING ABOUT IS HOW TO OPERATIONALIZE THE RESEARCH QUESTION.

EXAMPLE OF A (SOMEWHAT VAGUE) PROCESS EVAL. RESEARCH QUESTION

ARE THE CLIENTS ACTIVELY ENGAGED IN THE DAY PROGRAM?

SO NOW YOU ASK YOURSEL F THESE QUESTIONS: 1. WHAT DOES ACTIVELY ENGAGED LOOK LIKE?

2. HOW DO I OPERATIONALIZE IT? (WHAT DO PEOPLE WHO ARE ACTIVELY ENGAGED DO?)

3. WHAT DO I LOOK FOR?

4.HOW DO I MEASURE IT USING WORDS?

• POSSIBLE OPERATIONAL DEFINITIONS: 1. RATE OF DAILY ATTENDENCE , ABSENTEEISM AND TARDINESS BASED ON

AVAILABLE CLIENT RECORDS 2. OBSERVATIONS BY EVALUATOR OF VERBAL INTERACTIONS AMONG

CLIENTS AND STAFF DURING ACTIVITIES. 3. OBSERVATIONS BY EVALUATOR OF CLIENTS FOLLOWING DIRECTIONS AND

COMPLETING TASKS IN ACTIVITIES. 4. OBSERVATIONS BY EVALUATOR OF CLIENT-GENERATED QUESTIONS AND

STATEMENTS THAT ARE RELEVANT TO THE ACTIVITY AT HAND. 5. OBSERVATIONS BY EVALUATOR OF NUMBER OF CLIENTS WHO LEAVE

ACTIVITIES BEFORE THEY ARE OVER OR GO OUT FOR UNSCHEDULED CIGARETTE BREAKS.

6. IN-DEPTH INTERVIEWS WITH 3 RANDOMLY SELECTED CLIENTS WHICH ASK THE QUESTIONS a. HOW IMPORTANT IS THE PROGRAM IN YOUR LIFE? b. WHY OR WHY NOT? c. WHAT DO YOU LIKE ABOUT THE PROGRAM? DON’T LIKE? THE EVALUATOR WILL LOOK FOR EVIDENCE OF IMPORTANCE AND RELEVANCE TO CLIENTS LIFE, EVIDENCE OF ENJOYMENT WHEN DISCUSSING THE PROGRAM AND THE NUMBER OF POSITIVE STATEMENTS ABOUT THE STAFF AND PROGRAM.

7. A BRIEF SURVEY OF ALL STAFF ASKING THEM TO RATE 1. OVERALL LELVEL OF CLIENT PARTICIPATION BASED ON A FIVE POINT LIKERT SCALE.

8. NUMBER OF DOCUMENTED CLIENT OUTBURSTS, NON-COMPLIANCE OR NON-PARTICIPATION, BASED ON A RANDOM AUDIT OF CLIENT CHARTS FOR ONE WEEK

NOTICE THAT BY ANSWERING THOSE FOUR QUESTIONS, YOU CREATE YOUR

1. SOURCES OF DATA

2. METHODS OF DATA COLLECTION

3. MEASURES OF “ACTIVELY ENGAGED”.

4. Coding categories (more later)

• IN QUALITATIVE RESEARCH, IT IS IMPORTANT TO MAKE SURE YOU UNDERSTAND WHAT ALL OF THE TERMS IN THE RESEARCH QUESTION LOOK LIKE!!!

• COMMONLY CONFUSED TERMS– PARTICIPATION– IMPROVEMENT– ADEQUATELY PREPARED– HOW CAN WE REACH CLIENTS– DO CLIENTS FEEL HELPED– ARE STAFF CARRYING OUT THE PROGRAM AS DESIGNED– ARE CLIENTS AND/OR STAFF “BUYING INTO” OR “INVESTED” IN THE

PROGRAM– WHAT ARE SOME OF YOURS?

IN QUALITATITIVE RESEARCH, YOU MUST BEGIN TO THINK ABOUT WHAT THESE MEAN AND WHAT TO LOOK FOR