Upload
deanne
View
27
Download
0
Tags:
Embed Size (px)
DESCRIPTION
26 th Annual Management Information Systems (MIS ) Conference February 14, 2013. National Assessment of Educational Progress (NAEP) Observable Data From Computer-Based Assessments Gregory Anderson, Ph.D. Senior Usability Expert Fulcrum IT, LLC. Outline. What are observable data? - PowerPoint PPT Presentation
Citation preview
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
1
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
26th Annual Management Information Systems (MIS) Conference
February 14, 2013
National Assessment of Educational Progress (NAEP) Observable Data From Computer-Based Assessments
Gregory Anderson, Ph.D.Senior Usability Expert
Fulcrum IT, LLC
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
2
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Outline
What are observable data?
What can we observe in the data?
How have we used what we’ve observed?
How can we observe more?
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
3
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Outline
What are observable data?
What can we observe in the data?
How have we used what we’ve observed?
How can we observe more?
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
4
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What are Observable Data?
Computer actions recorded into a database
Possible with Computer-Based Assessments (CBA)
Automatic recording of low level information – Initially
• Number of key presses• Number of word processing functions (cut, copy, etc.)• Button clicks (Next, Back, etc.)• Screen coordinates (mouse click)
– Recently added:• Time stamping• Extended data (keys pressed, etc.)
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
5
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What are Observable Data?CBA2011 (Math Computer Based Study MCBS Grade 8 – Adaptive)1 Enter Item2 Exit Item3 Change Volume4 Change Language5 Keypress Count6 Click Choice7 Click Progress Navigator8 Keyboard Cut9 Right Click Cut10 Keyboard Copy11 Right Click Copy12 Keyboard Paste13 Right Click Paste15 Text-To-Speech17 Collapse Prompt18 Expand Prompt19 Collapse Response20 Expand Response21 Zoom22 Admin Move-To-Next-Block23 Hide Timer24 Show Timer25 Change Skin26 Application Recovery28 Swap Mouse Button29 Play30 Pause
31 Reach End of Tutorial32 Backspace33 Delete34 Clear Answer35 Mouse Cut36 Menu Cut37 Mouse Copy38 Menu Copy39 Mouse Paste40 Menu Paste41 Mouse Bold42 Keyboard Bold43 Menu Bold44 Right Click Bold45 Mouse Italic46 Keyboard Italic47 Menu Italic48 Right Click Italic49 Mouse Underscore50 Keyboard Underscore51 Menu Underscore52 Right Click Underscore53 Mouse Indent54 Mouse Outdent55 Mouse SpellCheck56 Menu SpellCheck57 Mouse Misspellings
58 Misspellings Identified59 Misspellings Corrected60 Menu Thesaurus61 Right Click Thesaurus62 Thesaurus Replacement63 Item Scroll64 Response Scroll65 Stimulus Scroll
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
6
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What are Observable Data?CBA2012 (2012 WCBA Pilot Grade 4)66 Drop Choice67 Keypress68 Highlight Mode On69 Highlight Mode Off70 Highlight Text71 UnHighlight Text72 Eliminate Mode On73 Eliminate Mode Off74 Eliminate Choice75 UnEliminate Choice76 Mouse Select All77 Keyboard Select All78 Menu Select All79 Mouse Undo80 Keyboard Undo81 Menu Undo82 Mouse Redo83 Keyboard Redo84 Menu Redo85 Right Click SpellCheck86 Mouse Thesaurus87 Click Word Definition88 Display Reminder89 Pause Workstation90 Resume Workstation
91 Enter Student Control Panel92 Leave Student Control Panel93 Menu Indent94 Menu Outdent
TEL2012 (2012 TEL Tryout Grade 8)95 Close Word Definition
CBA2013 (2013 TEL Pilot Grade 8)96 Reset Booklet97 Next98 Back99 Yes100 No101 OK102 Media Interaction
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
7
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Event ID
Event Type ID
Time Stamp Booklet ID Item ID
Block ID
Extended Data a (misspelling)
Extended Data b (corrected to)
66784 59 21:40.2 4069 41 36 exersize exercise
68085 59 42:47.8 4069 35 10 begining beginning
68255 59 45:05.3 4069 35 10 ningas ninja
74559 59 39:41.2 4071 70 24 lemonad lemonade
74969 59 44:53.9 4071 70 24 fliyers fliers
75609 59 57:18.7 4071 70 24 arived arrived
75614 59 58:25.7 4071 70 24 haelp help
77638 59 28:15.8 4071 37 12 monkey.You monkey. You
Putting the “Data” in Observable Data
Event Type ID Event Type58 Misspelling Identified
59 Misspelling Corrected
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
8
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Observable Data Sources
2011 Writing Computer-Based Assessment Pilot – Grade 8: 24,281 Students– Grade 12: 28,238 Students
2011 Math Computer-Based Study– Grade 8: 8,427 Students
2012 Writing Computer-Based Assessment– Grade 4: 13,005 Students
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
9
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Outline
What are observable data?
What can we observe in the data?
How have we used what we’ve observed?
How can we observe more?
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
10
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Can We Observe?
Frequency/Proportion (event ID, booklet ID, total number of participants)
End State (event ID, booklet ID, order of events)
Time Course (event ID, time stamp)
Duration/Intervals (event ID, 2 time stamps, math)
Qualitative (event ID, extended data)
Big Picture Qualitative (all data, other data sources)
Easy
Difficult
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
11
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Can We Observe?
Frequency/Proportion (event ID, booklet ID, total number of participants)
End State (event ID, booklet ID, order of events)
Time Course (event ID, time stamp)
Duration / Intervals (event ID, 2 time stamps, math)
Qualitative (event ID, extended data)
Big Picture Qualitative (all data, other data sources)
Easy
Difficult
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
12
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Observing Frequency/Proportion
How many times did the student use the mouse to cut? How many times did the student use the [CTRL+X] cut keystroke combination? How many times did the student use the menu to cut? How many times did the student cut using the right-click context menu? How many times did the student use the mouse to copy? How many times did the student use the [CTRL+C] copy keystroke combination? How many times did the student use the menu to copy? How many times did the student copy using the right-click context menu? How many times did the student use the mouse to paste? How many times did the student use the [CTRL+V] paste keystroke combination? How many times did the student use the menu to paste? How many times did the student paste using the right-click context menu? How many times did the student use the mouse to click the bold button? How many times did the student use the [CTRL+B] bold keystroke combination? How many times did the student use the menu to boldface? How many times did the student boldface using the right-click context menu? How many times did the student use the mouse to click the italicize button? How many times did the student use the [CTRL+I] italic keystroke combination? How many times did the student use the menu to italicize? How many times did the student italicize using the right-click context menu? How many times did the student use the mouse to click the underline/underscore button? How many times did the student use the [CTRL+U] underline/underscore keystroke combination? How many times did the student use the menu to underline/underscore? How many times did the student underline/underscore using the right-click context menu? How many times did the student use the mouse to click the spell check button? How many times did the student use the menu to check spelling? How many times did the student check spelling using the right-click context menu? How many times did the student right-mouse click misspelled words? How many times did the spell check identify misspellings?
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
13
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Scrolled Help Zoom Highlight Txt2Spch ShowTimer Hide Timer0%
10%20%30%40%50%60%70%80%90%
100%
70%
15%
33%
47%
72%
53%56%
60%
7%
16%
32%
53%
34% 36%
WCBA 2011System Tools
Grade 8 (n=24,281)
Grade 12 (n=28,238)
Perc
ent o
f Stu
dent
s
Available: WCBA 2010
Observing Frequency/Proportion
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
14
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Proportion Told Us
All tools were triggered at least once by a higher percentage of 8th graders than 12th graders
Text-to-Speech and scrolling were used by a greater percentage of students than other system tools
Help was used by a smaller proportion of students than other system tools
What Proportion Doesn’t Tell Us
Which students used multiple tools
If tools were used in some sections more than others
When and for how long each tool was used
The impact of tool use on performance– Exploration vs. need
Observing Frequency/Proportion
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
15
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Can We Observe?
Frequency/Proportion (event ID, booklet ID, total number of participants)
End State (event ID, booklet ID, order of events)
Time Course (event ID, time stamp)
Duration/Intervals (event ID, 2 time stamps, math)
Qualitative (event ID, extended data)
Big Picture Qualitative (all data, other data sources)
Easy
Difficult
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
16
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Grade 8Blue–72%High Contrast–21% Gray–7%
Observing End StateAvailable: WCBA 2010
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
17
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Observing End State
What End State Told Us
Blue was the end state skin for 72% of participants
Grey was the least used (7%)
What End State Doesn’t Tell Us
When they changed to the final state (beginning? end?)
How long they spent in the end state
Why they changed
How many blue users surveyed the other skins first (i.e., kept the default vs. choosing it)
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
18
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Can We Observe?
Frequency/Proportion (event ID, booklet ID, total number of participants)
End State (event ID, booklet ID, order of events)
Time Course (event ID, time stamp)
Duration/Intervals (event ID, 2 time stamps, math)
Qualitative (event ID, extended data)
Big Picture Qualitative (all data, other data sources)
Easy
Difficult
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
19
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Available: MCBA 2011Booklet #6037
Observing Time Course
10 min 13 min 16 min 19 min 22 min 25 min 28 min 31 min0
5
10
15
20
25
30
35
Timeline - Item 71
Corr
ecte
d M
issp
ellin
g Ev
ent #
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
20
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Time Course Told Us
When Corrected Misspelling events occurred
How long between Corrected Misspelling events
What Time Course Doesn’t Tell Us
Which words were misspelled
Number of words that were not identified as misspelled
If spellcheck was used to good effect
What was happening in between spellcheck events
Observing Time Course
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
21
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Can We Observe?
Frequency/Proportion (event ID, booklet ID, total number of participants)
End State (event ID, booklet ID, order of events)
Time Course (event ID, time stamp)
Duration/Intervals (event ID, 2 time stamps, math)
Qualitative (event ID, extended data)
Big Picture Qualitative (all data, other data sources)
Easy
Difficult
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
22
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Duration(2 time stamps, math)
Observing Duration/Intervals
VE521278c16 VE521279n160
3
6
9
12
15
1817.5
15.16
MCBS 2011Average Time Spent on Task Directions
Seco
nds
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
23
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Duration(2 time stamps, math)
Observing Duration/Intervals
Students Senior Usability Expert
05
101520253035404550
17.5
47
MCBS 2011Average Time Spent on These Di-
rections
Seco
nds
YOU ARE PERMITTED TO USE A CALCULATOR IN THIS SECTION.
This section has 16 questions. Respond to questions on the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded.
Remember:
• You are permitted to use a calculator in this section; either your own, or the one provided by NAEP. If you need help using the one provided by NAEP, read the instructions on the back of the scratch paper or look at the quick reference card on the calculator lid.
• If the calculator does not work or you do not know how to use it, raise your hand and ask for help.
• You will have to decide when to use the calculator. For some questions using the calculator is helpful, and may even be necessary. However, for other questions the calculator may not be helpful.
• This section is 25 minutes. If you finish before time is up, you may review any question within this section. You will not be able to go to the next section until 25 minutes are up.
• You may use the scratch paper provided, but be sure to enter all of your responses on the computer.
Click NEXT to continue.
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
24
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Duration(2 time stamps, math)
Observing Duration/Intervals
Students Senior Usability Expert
05
101520253035404550
15.16
27
MCBS 2011Average Time Spent on These Di-
rections
Seco
nds
YOU ARE NOT PERMITTED TO USE A CALCULATOR IN THIS SECTION.
This section has 16 questions. Respond to questions on the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded.
Remember:
• You are NOT permitted to use a calculator in this section.
• This section is 25 minutes. If you finish before time is up, you may review any question within this section. You will not be able to go to the next section until 25 minutes are up.
• You may use the scratch paper provided, but be sure to enter all of your responses on the computer.
Click NEXT to continue.
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
25
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Duration Tells Us
Students spent from 15-20 seconds on the directions for different blocks
Students on average spent more time on the directions for VE521278c16
Senior Usability Experts are really slow readers!
What Duration Doesn’t Tell Us
What students were doing while on that page
How much of the directions they read
How well they understood the directions
Observing Duration/Intervals
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
26
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Can We Observe?
Frequency/Proportion (event ID, booklet ID, total number of participants)
End State (event ID, booklet ID, order of events)
Time Course (event ID, time stamp)
Duration/Intervals (event ID, 2 time stamps, math)
Qualitative (event ID, extended data)
Big Picture Qualitative (all data, other data sources)
Easy
Difficult
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
27
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Observing Qualitative Information
No Correction 7%
Spelling 73%
Spacing 20%
Spelling Examples (includes punctuation)– “befor” vs. “before”– “ninjas” vs. “ninja’s”
Spacing Examples– “monkey.You” vs. “monkey. You”– “books,none” vs. “books, none”
No Correction Examples– “stonger,faster,and” (no alternate spelling
suggested)– “orrhjrejfjhhrhgjshjdnewjmfnffjnendndsms
andbfnnwnnfgbnemsnfcnfnsehgfnbsjehdnbvfndbfjxdnnebndfnfbfbfbbdbdnbsnsnnnsnsnsnnssnsnnsnnsnssssssssdjejkwjennbrbbfbbfbfcbfcbcvbbcxbcxblolooooooooooooooooooooooobbxbbx”
Identified Error Types
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
28
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Qualitative Told Us
What was modified
How it was modified
When it was modified
What It Didn’t Tell Us
The impact of the modification on the final output
Re-creation of the final product
Observing Qualitative Information
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
29
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Outline
What are observable data?
What can we observe in the data?
How have we used what we’ve observed?
How can we observe more?
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
30
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Major Takeaways
Students used accommodations and system and WP tools substantially during CBAs.– Scrolling: 60%-70%– Text-to-Speech: 53%-72%
Feature or Accommodation
4th Grade
8th Grade
12th Grade
Spanish 55% 56%
Text-to-Speech 82% 72% 53%
Show Timer 72% 53% 56%
Hide Timer 78% 34% 36%
Zoom 64% 33% 16%
Scrolled 63% 70% 60%
Help 22% 15% 7%
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
31
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Major Takeaways
Some accommodations and tools were not used as might be expected– Accommodation – Directions written in Spanish– Used by - 55%-56% of eligible students
Used56%
Did Not Use44%
Grade 12
Used55%
Did Not Use45%
Grade 8
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
32
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Major Takeaways
Observable data helped to identify and validate usability issues with CBAs.
Misuse of Next button– To open writing panel– To check on time remaining
Pausing tutorials (only 8%-9% of pausers didn’t finish)
Scrolling was not used by 7%-14% of participants on items where it was necessary in order to show all the information.
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
33
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Major Takeaways Insight into student response patterns
– Number of words used for different writing purposes
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
34
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Major Takeaways Insight into student response patterns
– Types of errors identified by spell check
No Correction 7%
Spelling 73%
Spacing 20%
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
35
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Major Takeaways Insight into student response patterns
– Distribution of spellcheck corrections by student
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
36
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Major Takeaways
MCBA Participant Average
Senior Usability Expert
100 200 300 400 500 600 700 800
Short 515
Short 289
Long 761
Long 283
National 204
National 280
Words per Minute
Insight into student response patterns– Compare silent reading rates
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
37
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
With Calculator Without Calculator YOU ARE NOT PERMITTED TO USE A CALCULATOR IN
THIS SECTION. This section has 16 questions. Respond to questions on
the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded.
Remember:• You are NOT permitted to use a calculator in this section.• This section is 25 minutes. If you finish before time is up,
you may review any question within this section. You will not be able to go to the next section until 25 minutes are up.
• You may use the scratch paper provided, but be sure to enter all of your responses on the computer.
Click NEXT to continue.
Major Takeaways
YOU ARE PERMITTED TO USE A CALCULATOR IN THIS SECTION.
This section has 16 questions. Respond to questions on the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded.
Remember:• You are permitted to use a calculator in this section; either your
own, or the one provided by NAEP. If you need help using the one provided by NAEP, read the instructions on the back of the scratch paper or look at the quick reference card on the calculator lid.
• If the calculator does not work or you do not know how to use it, raise your hand and ask for help.
• You will have to decide when to use the calculator. For some questions using the calculator is helpful, and may even be necessary. However, for other questions the calculator may not be helpful.
• This section is 25 minutes. If you finish before time is up, you may review any question within this section. You will not be able to go to the next section until 25 minutes are up.
• You may use the scratch paper provided, but be sure to enter all of your responses on the computer.
Click NEXT to continue.
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
38
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Written Instructions?
Written instructions are not being read for comprehension
Possible Causes / Consequences– The students suffered for not having read the
instructions– The students didn’t need the instructions– Some students didn’t need the instructions and
skewed the average duration data
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
39
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
What Do We Know?
How students use system tools
How students use word processing tools
What keys they are pressing and when
What buttons they are clicking and when
What words are identified by spell check, which are corrected and how they are corrected
How much time students spend on each screen
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
40
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Outline
What are observable data?
What can we observe in the data?
How have we used what we’ve observed?
How can we observe more?
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
41
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Descriptive vs. Experimental
Observable Data are by definition descriptive
Don’t answer causation or other complex questions
Causation needs experimental data– Isolate variables– Manipulate conditions– Compare groups
Complex questions– Pair Observable Data with other information
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
42
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
The More You Know, The Less You Know
Simple Question: Can Students Access the Features of CBA?– Simple Answer: Yes
Complex Questions– Did the changes made to interface have any impact on their
performance or comfort during assessment?
– Did everyone who would have benefited use them?
– Do kids even know what benefits them?
– What is impact of age/grade on beneficial impact of interface changes?
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
43
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
More Questions
Why do some students not use system tools?
Why do some students not use word processing tools?
Which tools result in better performance?
Are there other tools that might prove beneficial?
Do written instructions help? Everyone?
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
44
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Better Questions = Better Answers
1. Start with questions to be answered
2. Determine sources of information necessary to answer question (research design)
– Obs. Data (OD) + Usability testing
– OD + Background Question Answers
– OD + Performance Scores
– OD + Qualitative Analyses
3. Interpret interactions among data
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
45
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Data Sources
Observable Data (behavior)– Easy to collect (now)– Every participant (not just a sample)– Many, many parameters (many, many)– Single source
Profile Data (outcomes, background)– Also easy to collect on every participant– Fewer parameters– Multiple sources
Qualitative Data (impact, usability)– Requires interpretation– Difficult to collect– Requires sampling (research design)
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
46
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Data: The Final Frontier
Level I – CountingSingle variable
Frequency /Proportion
Level II – StructureExtended information
Time course / Duration
Level III – MeaningMultiple variables
Interactions among variablesQualitative information
ValueValue
Fulcrum IT Services Company | 7403 Gateway Court | Manassas, VA 20109 703.331.1500 | http://www.fulcrumit.com
47
Fulcrum IT Services, LLC | 5870 Trinity Parkway, Centreville, VA 20120 703.543.0000 | http://www.fulcrumit.com
Fulcrum IT Services, LLC
National Assessment of Educational Progress (NAEP) Observable Data From Computer-Based Assessments
Gregory Anderson, Ph.D.Senior Usability Expert
Fulcrum IT, LLC