View
218
Download
2
Category
Tags:
Preview:
Citation preview
AAER ConferenceNov 04 2010
Symposium onLimitations of Scientific Knowledge in Educational Research: Bias, Non-Significant Findings & Knowledge Representation
Jaya KannanLarry Lutsky
Yasmine AlwanLisa Bauer
Non-Significant Findings and a Research Response in Self-Directed Learning
Analyzing students’ articulation of learning goals:
How Problems in Research Led to Solutions in Teaching
Jaya KannanLarry Lutsky
Yasmine AlwanLisa Bauer
Purpose of the presentation
Share research work on LGsDiscuss rubric developmentExamine what didn’t work and what
was learnedElicit feedback from participants
Brief History of Learning GoalsAndragogy: theories of adult learning
(Knowles)Learning goals individualize learning &
contribute to the development of self-directed learning
The Purpose Centered Model
Relevance of Learning Goals Constructive Action teaching practicesSeeing connections in goal setting in
different contexts
Evolution of the RubricNeed to design a rubric Method of working – cyclicalEstablishing inter-rater reliability
Scale items Operational definitionsMath and writing – learning contextsExamples for each scale
Evolution of the rubricDisagreements/clarification:Defining SpecificityDefining complexityWhat constitutes measurability?
Should we use a zero scale?
Evolution of the rubric
Resolutions• Defining Specificity
– Break it up into content and syntax
• Defining complexity– Establishing parameters for what is/is not a
multi-layer goal –case by case evaluation
• Should we use a zero scale?– Only for measurability
• What constitutes measurability?– Arrived at keywords
Rubric – early version1 2 3
Specificity Not specific – I want to learn math
Moderately specific – I want to understand mathematical concepts
Highly specific – I want to learn basic algebra. I want to get an “A” in my course.
Complexity Low complexity – I want to learn how to multiply and divide fractions.
Moderate complexity – I want to learn about statistics so I
can analyze the data from my CA
Highly complex – I want to learn algebra so that I can apply the concepts in my course and also apply outside the class room.
Measurability Low measurability – I want to have more confidence in my ability.
Moderate measurability – I want to improve my math skills
High measurability – I want to be able to solve quadratic equations. I
want to be able to calculate the mean and standard deviation. I want
to be able to determine if there is subject-verb agreement
Rubric – final version1= Low 2 3 = High
Specificity –Content
Specificity - Syntax
I want to learn math.I want to understand mathematical concepts.Broad area superset
I want to learn basic algebraLess broad subset
I want to solve algebraic equationsNarrow subset
I want to understand fractionsDescribes an action
To improve my math skills by completing math problemsDescribes an action in detail
To learn sign numbers so I can solve equations correctlyDescribes more than one action with detail
Complexity I want to learn fractions1 layer of goal
I want to learn how to multiply and divide fractions
>=2 layers of goal
I want to learn about statistics so I can analyze data from my CA
>= 2 layers of goal with application
Measurability I want to improve my math skillsAny desirable changeStated demonstration of ability
I want to reduce the numbers of errors I make while solving math equations by the end of the semesterSomething quantifiable
By the end of the semester, I want to reduce the number of errors I make and complete more work
independently. 2 layers - Quantifiable, time frame
Inter-Rater Agreement(N = 4 raters)
94.4%97.2%86.1%
• Percent of Pairs of Raters within One Score Point of Each Other
58.3%58.3%47.2%• Percent of Pairs of Raters
with Exact Agreement
0.470.670.67
• Mean Inter-Rater Difference
MeasurabilityComplexitySpecificityAgreement Measure
Results of data collected
Results of data collected
Overall change in means over semesters Change in first 2 means, divided by # of sessions
Flaws/Challenges in MethodologyApplying hypothesis to problematic data
Retroactive analysis Abundant data Edu purposes
Lack of standardization Goals Raters
Measurement Definitions of scale itemsConsensus reached through extensive
discussion
What was learned
Standardization is difficult to achieve in teaching practiceShould it be standardized in the first
place? Limits in student knowledge and skillsActual work vs. projected work
Teaching styles
Follow up: Next Steps
Discussions with specialists improve understanding
(using the rubric)uniformityRevising goal writing formBuilding in measurability
Generating templates of goals
Recommended