MHS 7740

Research in Counseling

Fall, 2007 Larry C. Loesch, Ph.D., NCC University of Florida Department of Counselor Education 1215 Norman Hall P.O. Box 117046 Gainesville, FL 32611-7046 (352) 392-0731, ext.225 lloesch@coe.ufl.edu

2 II. Course Prerequisites A course in basic statistics (e.g., STA 3013, STA 3023, or equivalent) and a course in basic principles of assessment (e.g., MHS 6200 or equivalent) are prerequisites to this course. Students who have not completed successfully these prerequisites prior to enrollment in the course should consult with me prior to official registration for this course. III. Course Goals and Objectives This course is a curricular requirement for all students in all programs in the U.F. De-partment of Counselor Education. The universality of this requirement reflects the belief of the department faculty that having basic and applied research, program evaluation, and associated measurement and evaluation skills is crucial for all counseling professionals. Discovery of new knowledge, better understanding of current practices, and effective professional communications are essential to further development and enhancement of the counseling profession. Accord-ingly, all counseling professionals are expected to have and be able to apply research and related skills. In addition, counseling professionals are by definition expected to be accountable for their professional and programmatic activities and practices. Therefore, substantive research, program evaluation, and related knowledge and skills are essential to fulfilling all the demands and expec-tations for effective professional functioning. The diversity of the professional interests of the students in the class affords you the op-portunity to become familiar with the wide variety of professional situations in which research skills may be applied effectively. However, such diversity also dictates the manner in which the course content may be conveyed. That is, the individualized nature of each student's interests necessitates that general principles of research be presented in class, with specific applications left to the responsibility of each student. Therefore, a significant portion of the benefits you (may) derive from the course is contingent upon your efforts. Please also remember that the class consists of students with widely varying backgrounds in research, statistics, measurement, and evaluation. Although a course in basic principles of psychological and educational assessment and a course in basic statistics are prerequisites to this course, it is acknowledged that students in the course have varying degrees of pertinent expertise when this course begins. Thus, you are encouraged to ask questions as desired or necessary, no matter what the level of sophistication of the question. And please be tolerant of any question asked, regardless of your personal level of expertise. Purposes There are two major purposes for this course, including to: (a) enhance your ability to comprehend pertinent professional literature, particularly in regard to interpretation of quantita-tive and qualitative research and (b) enable you to develop basic research and evaluation skills. A related purpose is to enhance your understanding of the factors that influence how research is presented in the professional literature. Of course, it is also a purpose of this course to enable you to understand how research and evaluation are integral to effective functioning in the coun-seling profession. Objectives As a result of successful completion of this course you will have knowledge of:

3 a. perspectives of the role of research in counseling and development.

b. basic types of research and their associated methodologies. c. basic statistical principles and procedures.

d. principles of research report development. e. general principles of program evaluation. f. guidelines, sources of assistance, and practices in professional publication.

g. principles of assessment instrument development. h. general principles of needs-assessment practices.

i. ethical and legal guidelines related to research. j. uses of computers for research purposes.

Also as a result of successful completion of this course you will have skills to:

a. explain different types of research and research methodologies. b. apply basic statistics to practical situations.

c. apply general principles of program evaluation. d. apply general principles of needs assessments.

e. develop assessment instruments appropriate to your professional activities. f. apply ethical and legal considerations in research to practical situations.

More specifically, if you complete the course successfully, you should be better able to:

a. use appropriate writing styles, formats and guidelines. b. critically analyze professional research literature.

c. identify and use appropriate statistical analyses. d. converse effectively about research.

e. conduct research using appropriate legal and ethical standards. f. use effective assessment/evaluation techniques.

g. make contributions to the professional literature. IV. Applicable Professional Standards Addressed This course is designed to meet the professional standards listed following. Council for the Accreditation of Counseling and Related Educational Pro-grams (CACREP) (2002) Section II, Subsection K-8 RESEARCH AND PROGRAM EVALUATION - studies that provide an understanding of types of research methods, statistical analyses, needs assessment, and program evaluation, including all of the following:

a. the importance of research and opportunities and difficulties in conducting research in the counseling profession; b. research methods such as qualitative, quantitative, single-case designs, action re-search, and outcome-based research;

4 c. use of technology and statistical methods in conducting research and program evalua-tion, assuming basic computer literacy; d. principles, models, and applications of needs assessment, program evaluation, and use of findings to effect program modifications; e. use of research to improve counseling effectiveness; and f. ethical and legal considerations.

Portions of this course also fulfill in part the following CACREP Standards: Section II, Subsection J-8, PROFESSIONAL ORIENTATION:

b. professional roles and functions including similarities and differences with other types of professionals.

Section C, MARRIAGE AND FAMILY COUNSELING/THERAPY PROGRAM STANDARDS:

5. research and technical applications in marital, couple, and family counseling and ther-apy.

Section B, MENTAL HEALTH COUNSELING PROGRAM STANDARDS:

2. strategies for community needs assessment to design, implement, and evaluate mental health care programs and systems.

Section C, Subsection 1, SCHOOL COUNSELING PROGRAM STANDARDS:

a. use, management, analysis, and presentation of data from school-based information (e.g., standardized testing, grades, enrollment, attendance, retention, placement), surveys, interviews, focus groups, and needs assessments to improve student outcomes. b. design, implementation, monitoring, and evaluation of comprehensive developmental school counseling programs (e.g., the ASCA National Standards for School Counseling Programs) including an awareness of various systems that affect students, school, and home. c. implementation and evaluation of specific strategies that meet program goals and ob-jectives.

Section II, Subsection C, DOCTORAL STANDARDS FOR COUNSELOR EDUCATION AND SUPERVISION:

5. design and implementation of quantitative and qualitative research and methodology including univariate, multivariate, and single-subject design. 6. design and implementation of qualitative research, including grounded theory, ethno-graphic, and phenomenological methodologies. 7. models and methods of assessment and use of data.

V. Methods of Instruction The primary method of instruction for this course is lecture in conjunction with presenta-tions of a set of PowerPoint slides for each major topic area in the course. In addition, experien-

5 tial learning activities are included for some topic areas. Students’ use of the Internet and/or other electronic resources for learning outside of class is assumed and expected. VI. Course Assignments and Evaluations Please remember that students admitted to the University of Florida Graduate School in general, and to programs in the Department of Counselor Education in particular, are expected to perform academically at high levels because of the high program admissions standards. Accord-ingly, within the contexts of both University and Department philosophies, a grade of "C" is con-sidered to reflect minimally acceptable academic performance, a grade of "B" is considered to reflect (appropriately) average academic performance (because "average performance" for graduate students is expected to be relatively high), and a grade of "A" is considered to reflect outstanding academic performance in a course. Therefore, it is anticipated that only a very few (and hopefully no) students will receive a grade of "C," the vast majority of students will receive a grade of "B," and a few students will receive a grade of "A" in this course. Receipt of a grade of "B" from amongst a group of very talented students is not something about which to be dis-heartened; it is something about which to be proud. Specific Course Requirements As a result of effective and successful participation in the course, you should (a) have knowledge of the basic concepts, principles, and skills related to research in the counseling pro-fession and (b) be able to develop a workable plan for a research study that you could conduct. Therefore, the course assessment and evaluation activities (components) are directly related to these goals. Component 1 (80 points possible) Eight (8) in class quizzes, each having a maximum of ten (10) possible points, will be administered throughout the academic term. These quizzes will be administered unannounced; some may be administered at the beginning of the class period, some may be administered at the end of the class period, and some may be administered in the middle of the class period --- on whatever days it seems appropriate to administer them. Each quiz will contain 10 multiple-choice items, each having four response choices. One point will be awarded for each correct re-sponse. The items on each quiz may encompass any course material covered and any assigned readings prior to the time the respective quiz is given. Your quizzes will be scored soon after they are administered and returned to you at the next regularly scheduled class meeting. Please retain the (returned) answer sheets from each of your quizzes until the end of the course in the event a dispute arises as to your quiz point total. --------------------------------------------------------------------------------------------------------------------- Note bene: For Components 2 and 3, you may choose to work independently OR you may choose to work collaboratively with between one and three other students enrolled in the course this academic term. If you choose to work collaboratively with another student or other students, you must do so for both Components 2 and 3. The choice is irrevocable; you cannot collaborate on Component 2 and then not collaborate on Component 3, or vice versa. If you choose to col-laborate on these components, one point total will be assigned to each component submitted, and

6 that point total will be assigned to each student in the collaboration for the respective compo-nents. Please note that if you choose to collaborate on the assignments, I will not subsequently listen to, entertain, or otherwise attend or respond to complaints or concerns about who did (or more often did not) do what or otherwise did not behave in ways that were of your liking. Think carefully about this choice and decision! --------------------------------------------------------------------------------------------------------------------- Component 2 (20 points possible) Select a quantitative research article of interest to you. If you find the article in a journal, make a photocopy of the entire article. If you find the article “online” (i.e., in electronic format), print a copy of the complete article. Next, identify the relevant variables for the study (i.e., the specific variables for which data were gathered) and present them in a list that divides the rele-vant variables into those that are continuous and those that are discrete. Last, write the applica-ble and appropriate hypotheses for the study. Staple your lists of relevant variables and hypothe-ses to the copy of the article, write your name(s) on the top page, and submit all the pages to me. Component 3 (60 points possible) Select either a quantitative or qualitative research study of interest to you. The article selected must be different from the one selected for Component 2. If you find the article in a journal, make a photocopy of the entire article. If you find the article “online” (i.e., in electronic format), print a copy of the complete article. Remembering that an “evaluative critique” is ob-jective and includes both positive and negative commentary, write an evaluative critique of the article you selected. This task will be best accomplished by using the evaluative questions and/or criteria documents presented to you at the beginning of the course. Note that the docu-ments provided are guidelines; not all suggested evaluative questions or criteria listed on them will be applicable to every research article. Staple the copy of the article to your critique, put your name on the top page, and submit all the pages to me. Course Evaluation Criteria While it should be obvious at this level of your education, you are reminded that there is an important qualitative aspect to the evaluations of all papers submitted. That is, mere submis-sion of component papers does not fulfill completely the requirements; each component paper submitted must evidence high quality thought, preparation, and work. You also should be aware that, as a very general rule, there is an inverse relationship between the amount of "feedback" on papers during evaluation and the general level of the quality of the work submitted. Thus, as a very general rule, the more feedback given by me on or about your papers, the lower the evalu-ated quality of the paper submitted. You will be given a class handout entitled, “Suggestions for Writing Effectively.” The information on it is a compilation of grammatical mistakes commonly made by students that I have observed over my years of reviewing and evaluating students’ papers. It is essential that you review these suggestions carefully, and then strive diligently to avoid making the mistakes addressed in them as you compose your component papers. Being a competent counseling pro-fessional means being able to communicate effectively in writing. Now is the time to learn how to write effectively!

7 Components 2 and 3 will be evaluated in regard to the following criteria:

Content -- The degree to which the paper includes required elements, appropriate topical material, and appropriate references.

Presentation -- specifically with attention to:

Cogency -- The degree to which there is coherence and forcefulness in the presentation(s) of ideas, theoretical propositions, arguments, con-cepts, and/or procedures.

Clarity -- The degree to which the presentation of, and especially the idea-tional transitions among, ideas, constructs, concepts, propositions, procedures, or information is understandable and comprehensible.

Logic -- The degree to which there is readily apparent sequential validity in the presentations of ideas, propositions, constructs, or argu-ments.

Grammar -- The degree to which the manuscript adheres to correct use of the rules of English grammar, including spelling, punctuation, grammatical construction, and format.

Style -- The degree to which the presentation adheres correctly to the style and format requirements as specified in the current edition of the APA Publication Manual.

Preliminary Submission of Component Papers You may submit a component paper for preliminary review as many times as you may deem appropriate up to one (1) week before the actual due date for the paper. This is in-tended to be a service to enhance your work in the course. I will review the paper and provide feedback to you as quickly and as thoroughly as I am able. You are strongly encouraged to take advantage of this offer of assistance. It likely will result in a better evaluation of the final ver-sion of your paper(s). Final Submission of Component Papers Final versions of Component 2 and 3 papers are due at the specified assignment due dates and times. Component papers should be typed and double-spaced throughout. These component papers should evidence having been proofread effectively and be well-organized and “clean” in appearance. Component 2 and 3 papers may be submitted in either “hardcopy” or electronic form (i.e., as an attachment to an e-mail to me). Electronically submitted forms of Components 2 and 3 should be presented as “PC” (not Mac) Word or WordPerfect files. The same submission deadlines apply regardless of whether the papers are submitted in hardcopy or electronic form. Bonus Quiz A “bonus quiz” will be administered during Finals Week of the academic term during the regularly scheduled class meeting time and in the room where the class meets throughout the semester. Participation in this activity is optional. The bonus quiz has a maximum score of 20

8 points, is comprehensive in nature (i.e., covers topics addressed throughout the course), and contains 20 multiple-choice questions (each worth one point). Participation in this activity is for students who have not achieved sufficient points for a particular desired grade based on their cumulative results from the regular student performance evaluation procedures in the course. Grading Scale The initial grading scale for this course is: Grade of A 141 - 160 points total Grade of B 91 - 140 points total Grade of C 71 - 90 points total Grade of E 0 - 70 points total The minimum point totals for grades of A, B, or C may be lowered; they will not be raised. Grades of Incomplete No grades of Incomplete (I) are given for this course. You receive whatever grade you have earned (A, B, C, or E) at the time grades are submitted to the Registrar's Office at the end of the academic term. VII. Course Topics The "instructional pace" of this course is, to a great extent, contingent upon the composi-tion of the students in the class and their respective learning abilities, styles, and interests. Therefore, it is difficult to specify exactly what will be covered on particular days during the term. However, the topics listed below will be covered sometime during the term (although not necessarily in the order presented):

State-of-the-art of research in the counseling professions Legal, social, and ethical issues and practices in research

Methods for critical analysis of research Types of research: Qualitative and quantitative

Single-subject research designs (large-sample) Research designs and their validities

Research questions and hypotheses Treatment/intervention descriptions

Instrument development: Surveys and questionnaires Raters and ratings

Statistical analyses of relationships Statistical analyses of differences

Principles of program evaluation Principles of needs assessment

Writing for publication Using computer-based technologies for research and/or data analyses

9 VIII. Course Schedule The daily (course) class meeting schedule will be distributed to you as a separate hand-out. Please look it over carefully, and attend equally carefully to the various idiosyncracies of the schedule. The schedule is designed to fulfill the state and university’s minimum class meet-ing time requirement for a three-credit academic course. Please be on time to class and be pre-pared to stay the entire class meeting time. It will be necessary to use all of the class meeting time as indicated on the schedule distributed to you. IX. Course Resources The following textbook is required, has been ordered for the course, and is available from local bookstores and other sources:

Fraenkel, J.R., & Wallen, N.E. (2005). How to design and evaluate research in education (6th ed.). Boston: McGraw-Hill.

The following text is an optional (but very strongly recommended) text for the course:

Cherry, A.L., Jr. (2000). A research primer for the helping professions Methods, sta-tistics, and writing. Belmont, CA: Wadsworth/Thompson Learning.

In addition, you are encouraged to review and use the following resources as needed:

American Psychological Association. (2001). Publication manual of the American Psy-chological Association (5th ed.). Washington, DC: Author.

Szuchman, L.T. (2002). Writing with style APA style for counseling. Pacific Grove, CA: Brroks/Cole Thompson Learning.

Kazdin, A.E. (1995). Preparing and evaluating research reports. Psychological Assess-ment, 7(3), 228-237.

The following also may be good resources, especially for specific types of research and/or spe-cific techniques:

Airasian, P., & Gay, L.R. (2003). Educational Research Competencies for analysis and application (7th ed.). Columbus, OH: Merrill.

Bodgan, R.C., & Biklen, S.K. (1997). Qualitative research for education: An introduc-tion to theory and methods (3rd ed.). Boston, MA: Allyn and Bacon.

Gall, M.D., Borg, W.R., & Gall, J.P. (1996). Educational research An introduction (6th ed.). New York, NY: Longman.

Hadley, R.G., & Mitchell, L.K. (1995). Counseling research and program evaluation. Pacific Grove, CA: Brooks/Cole.

Heppner, P.P., Kivlighan, D.M., Jr., & Wampold, B.E. (1998). Research design in counseling (2nd ed.). Pacific Grove, CA: Brooks/Cole.

Houser, R. (1998). Counseling and educational research. Thousand Oaks, CA: Sage.

10 Leong, F.T., & Austin, J.T. (Eds). (1996). The psychology research handbook.

Thousand Oaks, CA: Sage. Stainback, S., & Stainback, W. (1988). Understanding & conducting qualitative re-

search. Dubuque, IA: Kendall/Hunt. Wilkinson, W.K., & McNeil, K. (1996). Research for the helping professions. Pacific

Grove, CA: Brooks/Cole. Any good, introductory (i.e., "basic") statistics text.

A comprehensive, collegiate-level dictionary. X. Course Policies Attendance There are no specific attendance requirements for this course and you are free to attend, or not attend, class meetings as you deem appropriate. However, over the history of my having taught this course it has become clear that there is a strong, positive relationship between class attendance and grades subsequently achieved in the course.... Please note that the class begins promptly at the scheduled time and that you should be to class on time so as not to disrupt the class meeting for others. Accommodations for Students with Disabilities If you have a disability or disabilities that require(s) some form of accommodation(s) in order for you to learn effectively, participate fully in this course, and/or to satisfy course assign-ments and/or requirements efficiently and effectively, you must contact me to discuss appropri-ate accommodation(s) for you by no later than the beginning of the third class meeting. Please be sure to bring a copy of your letter from the Office of the Dean of Students that attests to your disability(ies) for me for my records when we meet to discuss the appropriate accommodation(s) for you. University of Florida Honor Code As a result of having officially registered for this course, you have indicated intention to comply with the University of Florida Honor Code, specifically delineated as:

I understand that the University of Florida expects its students to be honest in all their academic work. I agree to adhere to this commitment to academic honesty and understand that my failure to comply with this commitment may result in dis-ciplinary action up to and including expulsion from the University.

If you are unwilling to abide by the University of Florida Honor Code, please withdraw from the course as soon as possible. Plagiarism Policy Plagiarism is defined as the presentation of a written work as if it were original, or for an original purpose, when in fact it is not. Engaging in the act of plagiarism is both unprofessional and a violation of the University of Florida Student Honor Code. If you present written work

11 for this course under the pretense that it is an original composition specifically and solely for this course when in fact it had been previously written by others or by you for another purpose, you are guilty of plagiarism. Also, submission of the same paper or other written work for con-current fulfillment of requirements in two courses is plagiarism within this definition. Note that plagiarism is distinctly different from reference citation or direct quotation from the work of an-other person or even from your own work; however, you must "give credit where credit is due" by providing accurate citations. I do not tolerate plagiarism. If it is determined that you have engaged in plagiarism, two things will happen automatically: (a) you will be assigned a grade of E for the course, and (b) a request will be made to the department chairperson to constitute a committee to determine your suitability for retention in the program in which you are enrolled. Course Drop Policy If it becomes necessary for you to "drop" this course, the University of Florida "Drop/Add" (i.e., course withdrawal) form must be signed by me; no other signature will be con-sidered valid for this purpose. Use of a Drop/Add form with a signature other than that of the course instructor is a violation of the University of Florida Student Honor Code. If you attempt to drop this course without my signature, two things will happen automatically: (a) you will be assigned a grade of E for the course, and (b) a request will be made to the department chairper-son to constitute a committee to determine your suitability for retention in the program in which you are enrolled. Consultation It will be easier for you to complete fully and successfully the course requirements with assistance beyond that provided in the class meetings. Accordingly, you should feel free, and indeed are encouraged, to arrange for mutually convenient individual consultation time with me. Further, you should feel free to request as many of these individual meetings as you believe are needed to complete course requirements effectively. Office Hours My office is in room 1212 Norman Hall, in the hallway opposite the main office for the Department of Counselor Education. My "official" office hours for this term are posted on the door to my office. In addition, I am often in my office and available to meet with you at times other than my "official" office hours. However, because of the many unforeseen demands on my time, it is a good idea for you to call or personally contact me so that we can arrange a mutually convenient meeting time. I keep my own calendar of meetings and the secretaries are often un-aware of my schedule of commitments; please do not call the office and ask the secretaries to set up an appointment for you with me. Telephone Numbers My office telephone number is (904) 392-0731, ext. 225. You may contact me at that number and/or leave a message to request a return call. The department number is 392-0731. Dial ext. 200 to contact a Department secretary. My home telephone number is (904) 377-7703. If you are unable to reach me at the of-fice, please feel free to call me at home between the hours of 10:00 a.m. - 8:00 p.m., Monday

12 through Friday. I would prefer that calls on Saturdays be restricted to "emergencies." Also, I take no responsibility for what I may do or say in response to course-related telephone calls after 8:00 p.m. or on Sundays.... My e-mail address is lloesch@coe.ufl.edu Generally, I check my e-mail early each morning, Monday through Friday. I only rarely check my e-mail later in the day or on week-ends, so please plan accordingly if you use this method to contact me.

Components of a Complete Research Report Chapter 1: Introduction a. Introductory lead b. Scope of the study (overview/background/context) c. Theoretical framework d. Statement of the problem e. Need for the study f. Purpose of the study g. Rationale for the methodology selected h. Hypotheses or research questions i. Definition of terms j. Overview of the remainder of the paper Chapter 2: Review of the Related Literature a. Further delineation of the problem (e.g., from I-b) b. Support for the need for the study (e.g., from I-d) c. Support for the theoretical framework (e.g., from I-f) d. Support for the approach to the study (e.g., from I-g) e. Support for the measurement techniques f. Summary of major points in Chapter II Chapter 3: Methodology a. Overview of the study b. Relevant variables c. Population d. Sampling procedures e. Resultant sample f. Research design (if applicable) g. Research procedures (if applicable) h. Treatment (if applicable) i. Measurement procedures j. Research participants (if applicable) k. Research participant training (if applicable l. Data analyses m. Methodological limitations Chapter 4: Results a. Presentation of data analyses b. Summary tables of data and results c. Nonevaluative explanation of results Chapter 5: Discussion a. Generalizability limitations b. Evaluations of hypotheses or research questions c. Conclusions and/or interpretations d. Implications e. Recommendations f. Summary of results and findings

Suggestions for Completing Component 2 1. List the relevant variables The “relevant” variables are those for which data are gathered – and that are “analyzed” in the data analyses for the study. Therefore, there are two good places in an article to examine to determine whether you have identified all the relevant variables for the study. One is in the methodology subsection (often in the description of the measurements). The other is in the re-sults subsection. Often the variables examined are included in data analysis tables presented. However, sometimes the results of data analyses are covered in the text. Check both. Be sure that you do not confuse “selection” variables with “relevant” variables. Some-times subjects are selected because they meet/fulfill certain criteria, such as being of a particular minimum age, being enrolled in a particular program, having specified characteristics, or even having achieved some criterion score on some measure. These “variables” are addressed to en-sure that subjects meet particular criteria for participation in the study. However, values of them are not analyzed as part of the data for the study, and therefore they are not “relevant” variables as the term ins being used in our context. 2. Divide the (relevant) variables into those that are continuous and those that are discrete. Confusion most frequently arises when the researchers/authors change a “naturally con-tinuous” variable into a discrete variable. For example, authors/researchers sometimes convert age to a discrete variable by subdividing the subjects into age groups. Similarly, they often change “income” into a discrete variable by creating “income categories.” And sometimes re-searchers/authors even mess around with response scales. For example, it is common to “weight” the response choices for a measurement instrument having Likert response scale (SA-A-U-D-SD) and then to sum the weights of the responses made to obtain a “score” on the in-strument. That’s okay, and yields a continuous variable. However, for reasons beyond my un-derstanding (although I have my suspicions), some authors/researchers will “combine” response categories (and the weights of response choices within them) into something categorical such as “favorable vs unfavorable” by combining SA and A for “favorable” and D and SD for “unfavor-able.” It’s not good research technique, but such nonsense is presented in may articles. 3. Relationship hypotheses. The general form for a relationship hypothesis is:

H∅N: There is no relationship between [continuous variable] and [(another) continuous variable].

The number of relationship hypotheses generated should encompass all the possible combina-tions. Most of the time (there are some rare exceptions), the number of possible relationship hy-potheses can be shown as a “triangular” matrix and determined from the formula: N x (N-1) x ½, where N equals the number of continuous variables for which hypotheses are being generated. For example, for a study having seven (7) continuous variables, the number of relationship hy-potheses should be 7 x 6 x ½ = 21. Note that it may be appropriate to create multiples of the above, usually depending on the discrete variables in a study (if any). For example, if for the preceding example, “gender (male or female) also was a variable, it likely would be appropriate to have 63 relationship hypotheses, i.e., 21 for the males, 21 for the females, and 21 for the total group. However, consideration of the situation is important. For example, for the example above, suppose the discrete variable was “treatment participation,” commonly expressed as experimental group and control group. In

such a situation, there would be a total of 42 appropriate hypotheses: 21 relationship hypotheses for the experimental group and another 21 for the control group. It would not be appropriate to consider the experimental and control groups combined. I’ll assume you can figure out why…. 4. Difference hypotheses The general form for a difference hypothesis is:

H∅N: There is no difference in the [continuous variable] based on (alternatively, as a function of) the [discrete variable].

The number of possible difference hypotheses generated should be the number of relationship variables times the number of discrete variables. 5. Interaction hypotheses Interaction hypotheses should be generated (only) if there is more than one discrete vari-able. If you want to find out how many interaction hypotheses there should be for a given num-ber of discrete variables in a study, Google “Pascal’s Triangle,” and then add one for the interac-tion that includes all the variables. For those among us who are mathematically-challenged, I suggest that creating a diagram is considerably easier…. For example, for a study having dis-crete variables A, B, and C, the interactions would include:

A-B* A-C* B-C* A-B-C**

* two-way interactions ** three-way interaction Similarly, for a study having discrete variables A, B, C, and D, the interactions would include:

A-B* A-C* A-D* B-C* B-D* C-D* A-B-C** A-B-D** A-C-D** B-C-D** A-B-C-D***

* two-way interactions ** three-way interactions *** four-way interaction

MHS 7740 -- Research in Counseling Guiding Questions for Evaluation of a Research Article

------------------------------------------------------------------------------------------------------------ The following questions were adopted from:

Pyrczak, F. (1999). Evaluating research in academic journals. Los Angeles: Pyrczak Publishing.

Do not cite or quote directly from the list following ------------------------------------------------------------------------------------------------------------ Is the title sufficiently specific? Did the title indicate the nature of the research without describing the results? If there is a main title and a subtitle, do both provide important information about the research? Did the title refer to the primary variables? Did the title indicate what types of people participated? Did the researcher avoid using jargon and acronyms that might be unknown to the readership? Is the purpose of the study presented, or at least implied, clearly in the title? Overall, was the title effective? Did the abstract highlight the research methodology? Has the researcher omitted the titles of measures in the abstract (except when they are the focus of the

research)? Were the highlights of the results described in the abstract? Did the researcher avoid making vague references to implications and future research? Overall, was the abstract effective? Was the lead to the topic interesting to the intended readership? Did the researcher begin by identifying a specific problem area? Did the researcher establish the importance of the problem area? Was the introduction an essay that flows logically from topic to topic? Did the researcher provide conceptual definitions of key terms? Did the researcher indicate the basis for (i.e., support) "factual" statements? Did the specific research purpose and related questions or hypotheses evolve logically from the in-

troductory material? Overall, was the introduction effective? If there was extensive literature on a topic, was the researcher been selective? Was the literature review critical? Was current research cited? Did the researcher draw literature from a variety of resources? Did the researcher provide a “balanced” presentation of the pertinent research? Did the researcher distinguish between research, theory, and opinion? Overall, was the literature review effective? What type of sampling was used in the research? If random sampling was used, what form of random sampling was used? If the sampling was impaired by refusals to participate, is the rate of participation rate sufficient?

If the sampling was limited by refusals to participate, is there reason and/or basis to believe that the par-ticipants and nonparticipants are similar on relevant variables?

What was the level or degree of representativeness of the sample? Did the researcher present sampling limitations? Did the researcher describe relevant demographics of the population and sample? Was the overall size of the sample adequate? Was there a sufficient number of participants in each subgroup that is reported on separately? Was informed consent obtained? Did the researcher describe the population in sufficient detail? Did the researcher describe the sample in sufficient detail? If a pilot study or developmental test of a theory, did the researcher use a sample with relevant demo-

graphics? If purposive sample was used, did the researcher indicate the basis for selecting individuals to include? Overall, was the sampling effective? Did the researcher provide behavioral definitions of key measurement constructs? Were actual measurement items, questions, and/or directions (or, at least, a sample of them) provided? Were any specialized response formats and/or restrictions described? For published instruments, were sources of additional information cited? If sensitive matters were studied / investigated, was there reason to believe that accurate data were ob-

tained? Were steps been taken to keep the instrumentation from obtruding on or changing overt behaviors

that were observed? If the collection and coding of observations was subjective, was there evidence that similar results would

be obtained if another researcher used the same measurement techniques with the same group at the same time?

If an instrument designed to measure a unitary trait was used, did it have adequate internal psycho-metric properties?

For presumably stable traits, was there evidence of the measurement’s temporal stability? If appropriate, was there evidence of the measurement’s content validity? If appropriate, was there evidence of the measurement’s empirical validity? Was the instrumentation adequate in light of the research purpose? Overall, was the measurement instrumentation / technique effective? If two or more groups were compared, were individuals assigned randomly to the groups? If two or more comparison groups were not formed at random, was there evidence that they were ini-

tially equal in important ways? If only a single participant or a single group was used, were the treatments alternated? Were the treatments described in sufficient detail? If the treatments were administered by people other than the researcher, were these people properly

trained? If the treatments were administered by people other than the researcher, was there a check to verify

that they were administered properly? If each treatment group had a different person administering a treatment, did the researcher try to

eliminate the "personal effect"? Except for differences in the treatments, were all other conditions the same for the experimental and

control groups? If necessary, did the researchers disguise the purpose of the experiment from the participants? Was the setting for the experiment "natural"?

Did the researcher use acceptable and ethical treatments? Did the researcher distinguish between random selection and random assignment. Overall, were the research procedures effective? Was the results section a cohesive essay? Did the researcher refer back to the research hypotheses, purposes, or questions originally stated in

the introduction? Were the statistical results presented in table form? If there were tables, were important aspects of them discussed in the narrative of the results sec-

tion? Did the researcher present descriptive statistics before presenting the results of inferential tests? If differences were statistically significant and small, did the researcher note that they were

small? Were appropriate statistical analyses used to analyze the data? Overall, was the presentation of the results effective? Did the researcher briefly summarize the purpose and results at the beginning of the discussion? Did the researcher acknowledge the methodological limitations? Were the results discussed in terms of the literature cited in the introduction? Did the researcher avoid citing new references in the discussion subsection? Were specific implications presented and discussed? Were suggestions for future research specific? Did the researcher distinguish between (personal) speculation and conclusions derived di-

rectly from the data? Overall, was the discussion effective? Did the researcher select an important problem? Was the researcher reflective? Is the article written cogently and cohesively? Did the article extend the knowledge on a topic? Were any major methodological flaws unavoidable or forgivable? Is the research likely to inspire additional research? Is the research likely to help in decision-making (either of a practical or theoretical na-

ture)? All things considered, is the report worthy of publication in an academic journal?

MHS 7740 -- Research in Counseling Guiding Questions for Evaluation of a Research Article

------------------------------------------------------------------------------------------------------------ The following was adopted from:

Gay, L.R., Mills, G.E., & Airasian, P. (2006). Educational research Competencies for analysis and applications (8th ed.). Columbus, OH: Pearson Merrill Prentice Hall.

Do not cite or quote directly from the following ------------------------------------------------------------------------------------------------------------ GENERAL EVALUATION CRITERIA Many research studies have flaws of various kinds. Just because a study is published does not nec-essarily mean that it is a good study or that it is reported adequately. The most common flaw is lack of va-lidity and reliability information about data-gathering procedures such as tests, observations, question-naires, and interviews. Other common flaws include weaknesses in the research design, inappropriate or biased selection of participants, failure to state limitations in the research, and a general lack of de-scription about the study. These common problems in studies reinforce the importance of being a com-petent consumer of research reports; they also highlight common pitfalls to avoid in your own research. At your current level of expertise, you may not be able to evaluate every component of every study. For example, you would not be able to determine whether the appropriate degrees of freedom were used in the calculation of an analysis of covariance. There are, however, a number of basic errors or weaknesses you should be able to detect in research studies. You should, for example, be able to identify the sources of invalidity associated with a study based on a one-group pretest-posttest design. You should also be able to detect obvious indications of experimenter bias that may have affected qualitative or quantitative research results. For example, a statement in a research report that "the pur-pose of this study was to prove. . ." should alert you to a probable bias effect. As you read a research report, either as a consumer of research keeping up with the latest find-ings in your professional area or as a producer of research reviewing literature related to a defined prob-lem, you should ask a number of questions about the adequacy of a study and its components. The answers to some of these questions are more critical than the answers to others. An inadequate title is not a critical flaw; an inadequate research plan is. Some questions are difficult to answer if the study is not directly in your area of expertise. If your area of specialization is reading, for example, you are probably not in a position to judge the adequacy of a review of literature related to anxiety effects on learning. And, admittedly, the answers to some questions are more subjective than objective. Whether a good design was used is pretty clear and objective; most quantitative researchers would agree that the randomized posttest-only control group design is a good design. Whether the most appropriate de-sign was used, given the problem under study, often involves a degree of subjective judgment. For ex-ample, the need for inclusion of a pretest might be a debatable point depending on the study and its de-sign. However, despite the lack of complete agreement in some areas, evaluation of a research report is a worthwhile and important activity. Major problems and shortcomings are usually readily identifiable, and by considering a number of questions, you can formulate an overall impression of the quality of the study. In the sections that follow, we list for your consideration evaluative questions about a number of research strategies and areas. This list is by no means exhaustive, and as you read it, you may very well think of additional questions to ask. You will also note that not every criterion applies equally to both quantitative and qualitative research studies.

Introduction Problem Is there a statement of the problem? Does the problem indicate a particular focus of study? Is the problem "researchable;" i.e., can it be investigated through collecting and analyzing data? Is background information on the problem presented? Is the educational significance of the problem discussed? Does the problem statement indicate the variables of interest and the specific relationship between

those variables that were investigated? When necessary, are variables directly or operationally defined? Did the researcher have the knowledge and skill to carry out the research? Review of Related Literature Is the review comprehensive? Are all cited references relevant to the problem under investigation? Are most of the sources primary (i.e., are there only a few or no secondary sources)? Have the references been analyzed and critiqued and the results of various studies compared and con-

trasted? That is, is the review more than a series of abstracts or annotations? Is the relevancy of each reference explained? Is the review well organized? Does it logically flow in such a way that the references least related to the

problem are discussed first and those most related are discussed last? Does it educate the reader about the problem or topic?

Does the review conclude with a summary and interpretation of the literature and its implications for the problem investigated?

Do the implications discussed form an empirical or theoretical rationale for the hypotheses that fol-low?

Are references cited completely and accurately? Hypotheses Are specific research questions listed or specific hypotheses stated? Does each hypothesis state an expected relationship or difference? If necessary, are variables directly or operationally defined? Is each hypothesis testable? Method Participants Are the size and major characteristics of the population studied described? Are the accessible and target populations described? If a sample was selected, is the method of selecting the sample clearly described? Does the method of sample selection suggest any limitations or biases in the sample? For example,

was stratified sampling used to obtain sample subgroups? Are the size and major characteristics of the sample described? If the study is quantitative, does the sample size meet the suggested guidelines for the minimum

sample size appropriate for the method of research represented? Instruments Do instruments and their administration meet guidelines for protecting human subjects? Were

needed permissions obtained? Is the rationale given for the selection of the instruments (or measurements) used? Are the purpose, content, validity, and reliability of each instrument described? Are the instruments appropriate for measuring the intended variables? Does the researcher have the needed skills or experience to construct or administer an instrument?

Is evidence presented to indicate that the instruments are appropriate for the intended sample? For example, is the reading level of an instrument suitable for sample participants? If appropriate, are subtest reliabilities given? If an instrument was developed specifically for the study, are the procedures involved in its devel-opment and validation described? If an instrument was developed specifically for the study, are administration, scoring or tabulating, and interpretation procedures fully described? Was the correct type of instrument used for data collection (or, for example, was a norm-referenced instrument used when a criterion-referenced one was more suitable)? Design and Procedure Are the design and procedures appropriate for examining the research question or testing the hy-

potheses of the study? Are the procedures described in sufficient detail to permit replication by another researcher? Do procedures logically relate to one another? Were instruments and procedures applied correctly? If a pilot study was conducted, are its execution and results described as well as its effect on the subse-

quent study? Are control procedures described? Does the researcher discuss or account for any potentially confounding variable that he or she was un-

able to control?

Results Are appropriate descriptive statistics presented? Was the probability level at which the results of the tests of significance were evaluated specified in

advance of the data analyses? Was every hypothesis tested? If parametric tests were used, is there evidence that the researcher avoided violating the required as-

sumptions for parametric tests? Are the described tests of significance appropriate, given the hypotheses and design of the study? Was the inductive logic used to produce results in a qualitative study made explicit? Are the tests of significance interpreted using the appropriate degrees of freedom? Are the results clearly described? Are the tables and figures (if any) well organized and easy to understand? Are the data in each table and figure described in the text? Discussion (Conclusions and Recommendations) Is each result discussed in terms of the original hypothesis or topic to which it relates? Is each result discussed in terms of its agreement or disagreement with previous results obtained by other

researchers in other studies? Are generalizations consistent with the results? Are the possible effects of uncontrolled variables on the results discussed? Are theoretical and practical implications of the findings discussed? Are recommendations for future action made? Are the suggestions for future action based on practical significance or on statistical significance only

(i.e., has the author avoided confusing practical and statistical significance)? Abstract or Summary Is the problem restated? Are the number and type of subjects and instruments described?

Is the design used identified? Are procedures described? Are the major results and conclusions restated? TYPE-SPECIFIC EVALUATION CRITERIA Descriptive Research Questionnaire Studies Are questionnaire validation procedures described? Was the questionnaire pretested? Are pilot study procedures and results described? Are directions to questionnaire respondents clear? Does each item in the questionnaire relate to one of the objectives of the study? Does each questionnaire item deal with a single concept? When necessary, is a point of reference given for questionnaire items? Are leading questions avoided in the questionnaire? Are there sufficient alternatives for each questionnaire item? Does the cover letter explain the purpose and importance of the study and give the potential respondent

a good reason for cooperating? If appropriate, is confidentiality or anonymity of responses assured in the cover letter? What is the percentage of returns, and how does it affect the study results? Are follow-up activities to increase returns described? If the response rate was low, was any attempt made to determine any major differences between respon-

dents and nonrespondents? Are data analyzed in groups or clusters rather than in a series of many single-variable analyses? Correlational Research Relationship Studies Were variables carefully selected (i.e., was a shotgun approach avoided)? Is the rationale for variable selection described? Are conclusions and recommendations based on values of correlation coefficients corrected for at-

tenuation or restriction in range? Do the conclusions avoid suggesting causal relationships between the variables investigated? Prediction Studies Is a rationale given for selection of predictor variables? Is the criterion variable well defined? Was the resulting prediction equation validated with at least one other group? Causal-Comparative Research Are the characteristics or experiences that differentiate the groups (the independent variable) clearly

defined or described? Are critical extraneous variables identified? Were any control procedures applied to equate the groups on extraneous variables? Are causal relationships found discussed with due caution? Are plausible alternative hypotheses discussed? Experimental Research Was an appropriate experimental design selected? Is a rationale for design selection given?

Are sources of invalidity associated with the design identified and discussed? Is the method of group formation described? Was the experimental group formed in the same way as the control group? Were groups randomly formed and the use of existing groups avoided? Were treatments randomly assigned to groups? Were critical extraneous variables identified? Were any control procedures applied to equate groups on extraneous variables? Were possible reactive arrangements (e.g., the Hawthorne effect) controlled for? Are tables clear and pertinent to the research results? Are the results generalized to the appropriate group? Single-Subject Research Are the data time constrained? Was a baseline established before moving into the intervention phase? Was condition or phase length sufficient to represent the behavior within the phase? Is the design appropriate to the question being asked? If a multiple baseline design was used, were conditions met to move across baselines? If a withdrawal design was used, are limitations to this design addressed? Did the researcher manipulate only one variable at a time? Is the study replicable?

Qualitative Research (In General) Does the topic studied describe a general sense of the study focus? Does the researcher state a "guiding hypothesis" for the investigation? Is the application of the qualitative method chosen described in detail? Is the context of the qualitative study described in detail? Is the purposive sampling procedure described and related to the study focus? Is each data collection strategy described? Is the researcher's role stated (e.g., nonparticipant observer, participant observer, interviewer, etc.)? Is the research site and the researcher's entry into it described? Were the data collection strategies used appropriately, given the purpose of the study? Were strategies used to strengthen the validity and reliability of the data (e.g., triangulation)? Is there a description of how any unexpected ethical issues were handled? Are strategies used to minimize observer bias and observer effect described? Are the researcher's reactions and notes differentiated from descriptive field notes? Are data coding strategies described and examples of coded data given? Is the inductive logic applied to the data to produce results stated in detail? Are conclusions supported by data (e.g., are direct quotations from participants used to illustrate points

made)? Interview Studies Were the interview procedures pretested? Are pilot study procedures and results described? Does each item in the interview guide relate to a specific objective of the study? When necessary, is a point of reference given in the guide for interview items? Are leading questions avoided in the interview guide? Is the language and complexity of the questions appropriate for the participants? Does the interview guide indicate the type and amount of prompting and probing that was permitted? Are the qualifications and special training of the interviewers described?

Is the method used to record responses described? Did the researcher use the most reliable, unbiased method of recording responses that could have been

used? Does the researcher specify how the responses to semistructured and unstructured items were quanti-

fied and analyzed? Narrative Research Does the researcher provide a rationale for the use of narrative research to study the chosen phenome-

non? Is there a rationale for the choice of individual to study the chosen phenomenon? Does the researcher describe data collection methods and give particular attention to interviewing? Does the researcher describe appropriate strategies for analysis and interpretation (e.g., restorying)? Ethnographic Research Does the written account (the ethnography) capture the social, cultural, and economic themes that

emerged from the study? Did the researcher spend a "full cycle" in the field studying the phenomenon? Mixed Methods Did the study use at least one quantitative and one qualitative data research method? Is a rationale for using a mixed methods research design provided? Is the type of mixed methods research design stated? Is the priority given to quantitative and qualitative data collection and the sequence of their use de-

scribed? Was the study feasible given the amount of data to be collected and concomitant issues of resources,

time, and expertise? Did the study investigate both quantitative and qualitative research questions? Are qualitative and quantitative data collection techniques clearly identified? Are the data analysis techniques appropriate for the type of mixed methods design? Action Research Does the area of focus involve teaching and learning in the researcher's own practice? Does the researcher state questions that were answerable given the researcher's expertise, time, and re-

sources? Was the area of focus within the researcher's locus of control? Is the area of focus something the researcher was passionate about? Is the area of focus something the researcher wanted to change or improve upon? Does the researcher provide an action plan detailing the impact of the research

findings on practice? Evaluating Validity and Reliability in Qualitative Studies1 Threats to Internal Validity Did the researcher effectively deal with problems of history and maturation by documenting historical

changes over time? Did the researcher effectively deal with problems of mortality by using a sample large enough to mini-

mize the effects of attrition? Was the researcher in the field long enough to effectively minimize observer effects? Did the researcher take the time to become familiar and comfortable with participants? Were interview questions pretested?

Were efforts made to ensure intraobserver agreement by training interview teams in coding procedures? Were efforts made to cross-check results by conducting interviews with multiple groups? Did the researcher interview key informants to verify field observations? Were participants demographically screened to ensure that they were representative of the larger popula-

tion? Was data collected using different media (audio- and videotape, etc.) to facilitate cross-validation? Were participants allowed to evaluate research results before publication? Is sufficient data presented to support findings and conclusions? Were dependent and independent variables repeatedly tested to validate results? Threats to External Validity Were constructs defined in a way that has meaning outside the study's setting? Were both new and adapted instruments pretested to ensure that they were appropriate for the study? Does the researcher fully describe participants' relevant characteristics, such as socioeconomic struc-

ture, gender makeup, level of urbanization and/or acculturation, and pertinent social and cultural history?

Are researcher interaction effects addressed by fully documenting the researchers activities in the setting?

Were all observations and interviews conducted in a variety of fully described settings and with multiple trained observers?

Reliability Is the researcher's relationship with the group and setting fully described? Is all field documentation comprehensive, fully cross-referenced and annotated, and rigorously de-

tailed? Were observations and interviews documented using multiple means (written notes and recordings, for

example)? Was the interviewer's training documented, and is it described? Was the construction, planning, and testing of all instruments documented, and are they de-

scribed? Are key informants fully described, including information on groups they represent and their commu-

nity status? Are sampling techniques fully documented as being sufficient for the study?

STUDENT INFORMATION FORM

It may be that I have to contact you during this academic term in regard to course and/or related matters. Therefore, it would be particularly helpful to me if you would please provide the following information and please PRINT ! Name _____________________________________________________ Address _____________________________________________________ _____________________________________________________ _____________________________________________________ Home Phone (______)_________________________ Work Phone (______)_________________________ Other Phone (______)_________________________ E-mail _________________________________________________ NOTE BENE: This information will be used by me only (e.g., if it necessary for me to contact you during this semester) and this sheet will be discarded at the end of the semester UNLESS you would like the information in your student file in the department updated. Please indicate following whether the information in your student file in the department can be updated using the information provided on this sheet. Yes, update my student file No, do not use this information for other than the purposes of this course.