Educational Research: Getting Started

Making Life Betterthrough Research, Education & Healthcare TM

Steven B. GoldinChief of Hepatobiliary and Pancreatic

SurgeryAssociate Professor of SurgeryVice Chair of Surgical Education

Chief of Hepatobiliary and Pancreatic Surgery

NO DISCLOSURES EXCEPT:Yes, they really are my

photos

NO DISCLOSURES EXCEPT:Yes, they really are my

photos

Outline1. Research: Why do it?2. What is educational research3. Research: How to get started4. Features of rigorous studies5. Pitfalls – validity and reliability6.Surveys7.Ethics, funding, resources

ASE – “No discipline can call itself a profession that does not give top priority to research. More specifically, the status of a given profession depends on the adequacy of the research activities, and the application of research findings to routine professional activities.” (Mouly 1978)

ASE – “No discipline can call itself a profession that does not give top priority to research. More specifically, the status of a given profession depends on the adequacy of the research activities, and the application of research findings to routine professional activities.” (Mouly 1978)

Research – why do it?

Faculty advancement

Teaching and research separate academic surgeons from those in community practice

Develop a national prominence/reputation

May be a funding source

Faculty advancement

Teaching and research separate academic surgeons from those in community practice

Develop a national prominence/reputation

May be a funding source

Research- why do it?

Importance of clinical productivity – how do you find the time to do a research project?

Do research with your daily activities

Data collection is common for all courses and rotations.

medical studentsresidentsfaculty

Innovation – what makes an academic center academic. What additional data would need to be collected to begin a research project?

Importance of clinical productivity – how do you find the time to do a research project?

Do research with your daily activities

Data collection is common for all courses and rotations.

medical studentsresidentsfaculty

Innovation – what makes an academic center academic. What additional data would need to be collected to begin a research project?

Introduction

Innovation and evaluation form the backbone of research ideas and projects

Many research projects are done with students, residents, and faculty

Steady supply of potential participants

Innovation and evaluation form the backbone of research ideas and projects

Many research projects are done with students, residents, and faculty

Steady supply of potential participants

Introduction

Traditionally empirical research (bench top) was viewed as the most valid method. Other forms of research (historical, phenomenological, introspective, qualitative, etc.) contribute to “educational research”

Empirical research is characterized by direct observation, objectivity, falsification, replicability, parsimony and quantification (scientific method)

Parsimony is defined as the adoption of the simplest assumption in the formulation of a theory or in the interpretation of data, especially in accordance with the rule of Ockham's razor

Traditionally empirical research (bench top) was viewed as the most valid method. Other forms of research (historical, phenomenological, introspective, qualitative, etc.) contribute to “educational research”

Empirical research is characterized by direct observation, objectivity, falsification, replicability, parsimony and quantification (scientific method)

Parsimony is defined as the adoption of the simplest assumption in the formulation of a theory or in the interpretation of data, especially in accordance with the rule of Ockham's razor

Introduction

Occam’s razor (Ockham’s razor) - “entities should not be multiplied unnecessarily.” The explanation of any phenomenon should make as few assumptions as possible, eliminating those that make no difference in the observable predictions of the explanatory hypothesis or theory

When competing hypotheses are equal in other respects, select the hypothesis that introduces the fewest assumptions and postulates the fewest entities while sufficiently answering the question

Occam’s razor (Ockham’s razor) - “entities should not be multiplied unnecessarily.” The explanation of any phenomenon should make as few assumptions as possible, eliminating those that make no difference in the observable predictions of the explanatory hypothesis or theory

When competing hypotheses are equal in other respects, select the hypothesis that introduces the fewest assumptions and postulates the fewest entities while sufficiently answering the question

Introduction

Evidence based medicine – interventions should be guided by data from rigorous studies was described at least 15 years ago

Evidence-Based Medicine (EBM) – excellent framework for evaluating the quality of clinical research

EBM is not focused on educational research, but provides useful criteria for different categories of research. (Treatment efficacy, accuracy of diagnostic tests, clinical guidelines, prognosis and risk, overviews and meta-analysis)

Evidence based medicine – interventions should be guided by data from rigorous studies was described at least 15 years ago

Evidence-Based Medicine (EBM) – excellent framework for evaluating the quality of clinical research

EBM is not focused on educational research, but provides useful criteria for different categories of research. (Treatment efficacy, accuracy of diagnostic tests, clinical guidelines, prognosis and risk, overviews and meta-analysis)

Evidence Based Medicine

Many EBM concepts are applicable to educational research and should be familiar to all educational researchers

Medical educators, likewise, should teach based on evidence

Many EBM concepts are applicable to educational research and should be familiar to all educational researchers

Medical educators, likewise, should teach based on evidence

Evidence Based Medicine

Educational research is driven by the process of documenting the steps of a project

This process is the same for all studies including bench research, studying teaching methods, or student attitudes

Educational research focuses on describing and documenting the transmission of knowledge, competency, and understanding about which teaching methods are best and can lead to better practice

Educational research is driven by the process of documenting the steps of a project

This process is the same for all studies including bench research, studying teaching methods, or student attitudes

Educational research focuses on describing and documenting the transmission of knowledge, competency, and understanding about which teaching methods are best and can lead to better practice

Introduction

All research involves – asking a question and obtaining an answer in a systematic, thoughtful way

Scholarly teaching involves routinely asking questions about daily educational activities

Research is answering these questions and scholarship is the dissemination of the research results

All research involves – asking a question and obtaining an answer in a systematic, thoughtful way

Scholarly teaching involves routinely asking questions about daily educational activities

Research is answering these questions and scholarship is the dissemination of the research results

Making it Scholarly

Descriptive: study describes what is going on or what exists

Relational: study looks at the relationships between two or more variables

Causal: study is designed to determine whether one or more variables causes or affects one or more outcome variable

Descriptive: study describes what is going on or what exists

Relational: study looks at the relationships between two or more variables

Causal: study is designed to determine whether one or more variables causes or affects one or more outcome variable

Types of Studies

1. Ask the question – state clear goals (hypothesis and alternate hypothesis)

2. Prepare (literature review, discussions, conferences)

3. Specify the techniques or interventions4. Test methods and ways of measuring

impact5. Pilot test the process and refine process6. Select the best method to answer the

question

1. Ask the question – state clear goals (hypothesis and alternate hypothesis)

2. Prepare (literature review, discussions, conferences)

3. Specify the techniques or interventions4. Test methods and ways of measuring

impact5. Pilot test the process and refine process6. Select the best method to answer the

question

Research Steps

7. Do the study

8. Capture and analyze data

9. Interpret findings and determine significance of the outcome

10.Negative results can be meaningful depending on the question

11.Reflective critique (thoughtful discussion, think about the implications of the findings and develop the next steps)

12.Disseminate the findings (article, abstract, presentation, workshop)

7. Do the study

8. Capture and analyze data

9. Interpret findings and determine significance of the outcome

10.Negative results can be meaningful depending on the question

11.Reflective critique (thoughtful discussion, think about the implications of the findings and develop the next steps)

12.Disseminate the findings (article, abstract, presentation, workshop)

Research Steps

Large numbers of student participants

Multi-institutional enrollment of participants

Having a control or appropriate comparison group

Measuring an objective outcome

Measuring some outcome at least 1 month after the intervention

Conduct the intervention more than once

Estimate statistical power

Large numbers of student participants

Multi-institutional enrollment of participants

Having a control or appropriate comparison group

Measuring an objective outcome

Measuring some outcome at least 1 month after the intervention

Conduct the intervention more than once

Estimate statistical power

Features of Rigorous Studies

Measure objective outcomes including any evaluation other than self-assessment by students including:

Tests of knowledgeCourse grades Objective structured clinical examinations or observations of standardized patient

Interaction assessment of performance with a real patient such as graded observation of patient interactions or patient feedback

Objective clinical outcomes such as blood pressure controlPerformance with mannequins, computer simulations, or laboratory animals

Psychological inventories

Measure objective outcomes including any evaluation other than self-assessment by students including:

Tests of knowledgeCourse grades Objective structured clinical examinations or observations of standardized patient

Interaction assessment of performance with a real patient such as graded observation of patient interactions or patient feedback

Objective clinical outcomes such as blood pressure controlPerformance with mannequins, computer simulations, or laboratory animals

Psychological inventories

Features of Rigorous Studies

Language

Types of questions (descriptive, relational, causal)

Time in research (cross-sectional vs. longitudinal with repeated measures or time series)

Types of relationships (correlational vs. causal)

Patterns of relationships (none, positive, negative)

Variables and attributes (independent and dependent, exhaustive, mutually exclusive)

Hypothesis and null hypothesis

Types of data (quantitative vs. qualitative)

Language

Types of questions (descriptive, relational, causal)

Time in research (cross-sectional vs. longitudinal with repeated measures or time series)

Types of relationships (correlational vs. causal)

Patterns of relationships (none, positive, negative)

Variables and attributes (independent and dependent, exhaustive, mutually exclusive)

Hypothesis and null hypothesis

Types of data (quantitative vs. qualitative)

Foundation

Theoretically significant research questions are different from other types of questions.

A research question is not a question:

of fact. (How many residents passed the in-service or their qualifying examinations?)

about a specific sample of subjects. (Do my students perform better with open-ended rather than multiple choice questions?)

about things that can not be measured. (Do lucky students have more success than unlucky students?)

Theoretically significant research questions are different from other types of questions.

A research question is not a question:

of fact. (How many residents passed the in-service or their qualifying examinations?)

about a specific sample of subjects. (Do my students perform better with open-ended rather than multiple choice questions?)

about things that can not be measured. (Do lucky students have more success than unlucky students?)

The Hypothesis

1. seek new knowledge

2. use the scientific method

3. build upon existing scientific knowledge

4. pick up where existing theories and data end

5. lead to specific hypotheses which can be tested

6. have an answer deducible from the results and provide information applicable not only to the subjects studied, but to a broad, definable population of subjects and settings

1. seek new knowledge

2. use the scientific method

3. build upon existing scientific knowledge

4. pick up where existing theories and data end

5. lead to specific hypotheses which can be tested

6. have an answer deducible from the results and provide information applicable not only to the subjects studied, but to a broad, definable population of subjects and settings

A good research question should:

Education research quality is linked to reliability and validity, which are the two major criteria for the quality of the measurement

Reliability involves the quality of measurement and the consistency or repeatability of the measures

Theories of validity involve each step which are interdependent

Education research quality is linked to reliability and validity, which are the two major criteria for the quality of the measurement

Reliability involves the quality of measurement and the consistency or repeatability of the measures

Theories of validity involve each step which are interdependent

Reliability and Validity

Studies may be

reliable but not valid

valid but not reliable

neither reliable nor valid

both reliable and valid

Consulting with a research design and statistics expert to avoid issues that may impact study reliability or validity

Studies may be

reliable but not valid

valid but not reliable

neither reliable nor valid

both reliable and valid

Consulting with a research design and statistics expert to avoid issues that may impact study reliability or validity

Reliability and Validity

General Surgeon Shortage

Surveys - References

Sage Publications, Inc; 2nd edition (November 12, 2002), 10 volumes, 1434 pages

Gall, Borg, and Gall. Educational Research: An Introduction (8th edition). Allyn and Bacon. 2006

One of the most important areas of measurement in applied social research.

Two types

Questionnaires

Interviews

Construction of the Survey

Types of questions

Decisions about question content

Decisions about question wording

Decisions about response format

Question placement and sequence

One of the most important areas of measurement in applied social research.

Two types

Questionnaires

Interviews

Construction of the Survey

Types of questions

Decisions about question content

Decisions about question wording

Decisions about response format

Question placement and sequence

Surveys

Often done because users think they are simple and quick ways to publications

Survey design has fallen into disrepute among editors, reviewers and consumers of medical research because they have been:

poorly designed

poorly developed

poorly administered

poorly analyzed

Reinforces the reconceived notion that surveys are not worth doing

Like experimental research, good survey research is very systematic and rigorous

Often done because users think they are simple and quick ways to publications

Survey design has fallen into disrepute among editors, reviewers and consumers of medical research because they have been:

poorly designed

poorly developed

poorly administered

poorly analyzed

Reinforces the reconceived notion that surveys are not worth doing

Like experimental research, good survey research is very systematic and rigorous

Surveys

The purpose of the study must be supported by a review of the literature and understanding of the problem. Then develop the research question, which will guide the development of the content. If an item does not help you answer a research question, do not include it on the survey

What information do you want from respondents?attitudes/beliefs?knowledge?behavior?attributes (demographic characteristics)?

The purpose of the study must be supported by a review of the literature and understanding of the problem. Then develop the research question, which will guide the development of the content. If an item does not help you answer a research question, do not include it on the survey

What information do you want from respondents?attitudes/beliefs?knowledge?behavior?attributes (demographic characteristics)?

Surveys

Structure the questions carefully

Unless the sole purpose of the survey is to solicit suggestions, probe memories, clarify positions and or vent frustrations, limit the number of open-ended questions

Open ended questions make it difficult to construct meaningful variables for statistical analysis, are difficult to collate, and difficult to decipher

Structure the questions carefully

Unless the sole purpose of the survey is to solicit suggestions, probe memories, clarify positions and or vent frustrations, limit the number of open-ended questions

Open ended questions make it difficult to construct meaningful variables for statistical analysis, are difficult to collate, and difficult to decipher

Surveys

A questionnaire should:1. look attractive/appealing to complete2. have questions laid out and be organized in such a manner as to make them as easy to complete as possible3. be organized with items appearing in some logical sequence4. begin with those questions that the respondent is likely to see as useful to the purpose of the study, or with those that are easy to answer and non-threatening5. place important items that are key to the study near the beginning

Items that are interesting and clearly relevant to the study will increase response rate. Length has a small effect on response rate.

Pilot the instrument

A questionnaire should:1. look attractive/appealing to complete2. have questions laid out and be organized in such a manner as to make them as easy to complete as possible3. be organized with items appearing in some logical sequence4. begin with those questions that the respondent is likely to see as useful to the purpose of the study, or with those that are easy to answer and non-threatening5. place important items that are key to the study near the beginning

Items that are interesting and clearly relevant to the study will increase response rate. Length has a small effect on response rate.

Pilot the instrument

Surveys

Principle of voluntary participationInformed consentRisk of harm – physical or psychologicalConfidentialityPrinciple of anonymityPerson’s right to service (no treatment control group)Institutional Review Board (IRB)

Students and residents are considered a vulnerable populationSocial and Behavioral Institutional IRB vs.

Biomedical IRB

Principle of voluntary participationInformed consentRisk of harm – physical or psychologicalConfidentialityPrinciple of anonymityPerson’s right to service (no treatment control group)Institutional Review Board (IRB)

Students and residents are considered a vulnerable populationSocial and Behavioral Institutional IRB vs.

Biomedical IRB

Ethics

Most surgical education research is conducted without the benefit of research grants

ASE Foundation – CESERT program (www.surgicaleducation.com)

National Board of Medical ExaminersEmergency Medicine FoundationAgency for Health Care Research and QualitySagesAmerican College of Surgeons (www.facs.org) lists funding agencies that may be willing to fund educational research

Most surgical education research is conducted without the benefit of research grants

ASE Foundation – CESERT program (www.surgicaleducation.com)

National Board of Medical ExaminersEmergency Medicine FoundationAgency for Health Care Research and QualitySagesAmerican College of Surgeons (www.facs.org) lists funding agencies that may be willing to fund educational research

Funding

ASE Educational Research Committee

Institutional Department of Medical Education

Departments of psychology, education, sociology, and public health

Biostatistician

ASE – Surgical Education Research Fellowship Program (SERF)

ASE Educational Research Committee

Institutional Department of Medical Education

Departments of psychology, education, sociology, and public health

Biostatistician

ASE – Surgical Education Research Fellowship Program (SERF)

Resources

Clinical demands are taking on larger levels of importance

Protected time is a luxury

Research is required to advance academically

Educational research is a viable option even when faced with high clinical demands

Clinical demands are taking on larger levels of importance

Protected time is a luxury

Research is required to advance academically

Educational research is a viable option even when faced with high clinical demands

Conclusions

Many educational research projects have significant limitations

Insufficient sample sizesLack of generalizability when interventions are

assessed at only 1 institution or are conducted only once

Lack of appropriate control groupsUse of subjective and non-validated instruments to assess outcomesAssessing only short-term outcomes

Many educational research projects have significant limitations

Insufficient sample sizesLack of generalizability when interventions are

assessed at only 1 institution or are conducted only once

Lack of appropriate control groupsUse of subjective and non-validated instruments to assess outcomesAssessing only short-term outcomes

Conclusions

Educational research challenges often relate to measurement issues, bias and individual differences

An understanding of key educational concepts in psychology, sociology, anthropology, psychometrics, sociometrics, and statistics. Collaboration is key!

Assuming a significant question has been clearly articulated, the quality of the study will depend upon the research design, data analysis, interpretation and results reporting

Educational research challenges often relate to measurement issues, bias and individual differences

An understanding of key educational concepts in psychology, sociology, anthropology, psychometrics, sociometrics, and statistics. Collaboration is key!

Assuming a significant question has been clearly articulated, the quality of the study will depend upon the research design, data analysis, interpretation and results reporting

Conclusions

Research Methods Knowledge Base. William M.K. Trochim. http://www.socialresearchmethods.net. Free on the internet

Surgical Educators’ Handbook. Curriculum Committee and Members of the Association for Surgical Education. ASE website. http://www.surgicaleducation.com/mc/page.do

Research Methods Knowledge Base. William M.K. Trochim. http://www.socialresearchmethods.net. Free on the internet

Surgical Educators’ Handbook. Curriculum Committee and Members of the Association for Surgical Education. ASE website. http://www.surgicaleducation.com/mc/page.do

References

Educational Research: Getting Started

Making Life Betterthrough Research, Education & Healthcare TM

Steven B. GoldinChief of Hepatobiliary and Pancreatic

SurgeryAssociate Professor of SurgeryVice Chair of Surgical Education

Chief of Hepatobiliary and Pancreatic Surgery