C R E S S T / U C L A
Paper presented at AERANew Orleans—April 2000
Using Technology to Assess Students’ Web Expertise
Davina C. D. KleinCRESST/UCLA
Louise YarnallCenter for Technology in Learning, SRI International
Christina Glaubke CRESST/UCLA
C R E S S T / U C L A
C R E S S T / U C L A
2
Technology Focus: WWW
Over 1.5 billion Web pages currently available, increasing at rate of 1.9 million pages per day
At least 40% of U.S. classrooms linked to Internet With 9 million children using Internet at school
Most common classroom use for WWW is research 3rd most common use of computers at school
Has surpassed drill-and-practice software
Nearly 90% of teachers perceive classroom WWW access as valuable or essential for their teaching
C R E S S T / U C L A
3
Research Approach
Study experienced Web users
Identify set of measures to be used to assess students’ fluency with the World Wide Web
Web Expertise Assessment (WEA) Web-based, authentic performance assessment
Tasks require students to navigate through large information space searching for relevant information and bookmarking relevant findings
All measures logged and coded
C R E S S T / U C L A
4
WEA Search Task
Imagine you are learning about the U.S. presidents in your history class. Your teacher has asked you to write a report about what presidents said during their speeches when first elected to office. She has asked you to find out which presidents spoke of the importance of an educational system available to all without charge.
Use WEA to find this information for your report.
Find as many useful pages as you can.
Bookmark pages by clicking on the Add Bookmark button near the top of your screen.
You may bookmark as many useful pages as you think necessary.
*
C R E S S T / U C L A
11
WEA Background Questionnaire
Used to evaluate students’ WWW background knowledge Paper-and-pencil survey
Students rated statements on scale of 1 (“I really don’t agree”) to 5 (“I really agree”) e.g., “The information on the World Wide Web is not
very useful”
C R E S S T / U C L A
12
Participants
120 middle and high school students Students had strong technology
background
Students had access to WWW in class
Students were familiar with navigating the World Wide Web
C R E S S T / U C L A
13
Methods
Students completed WWW background questionnaire
Students trained on WEA
Students given 20 minutes to complete WEA search task Reminded to bookmark relevant pages
Data coded with high interrater reliabilities (ranging from .97-.99)
C R E S S T / U C L A
14
Evaluating WEA PerformanceItem Factor loadings
F 1 F 2 F 3 F 4Number of times back used .97Number of steps in search .92Number of revisited info pages .76Average bookmark score .92Quality of bookmark set .87Efficiency of search .85Web helpful with search .81Web info useful .80Web info correct .73Web info detailed .69Number of good searches .91Quality of keyword search set .77Number of redirected searches .66Eigenvalues 2.54 2.37 2.34 2.00Percent variance explained 19.5% 18.2% 18.0% 15.4%Note. Only factor loadings with absolute values of .30 and higher shown.
navigationalstrategiesα = .88
findingabilityα = .86
backgroundWeb
knowledgeα = .76
searchingexpertiseα = .71
C R E S S T / U C L A
15
Creating WEA Scales
Navigational strategies (α = .88) Number of times back button used Number of steps in search Number of revisited information pages
In general, students navigated well Students used back often (M = 17) for orientation Students revisited over one third of info pages
visited, orienting themselves in Web space (M = 4) Students completed many steps (M = 93)
C R E S S T / U C L A
16
Creating WEA Scales (con’t.)
Finding ability (α = .86) Average bookmark score Quality of bookmark set Efficiency of search
In general, students able to find information Average bookmark peripherally relevant to task
(M = 1.8 on 0-3 scale) Quality of bookmark response set good
(M = 1.8 on 0-3 scale) About one fifth of pages bookmarked
appropriately (efficiency M = .21)
C R E S S T / U C L A
17
Creating WEA Scales (con’t.)
Background Web knowledge (α = .76) WWW helpful in finding information Information on WWW not very useful (r) Information on WWW is accurate/correct Not a lot of detailed information on WWW (r)
In general, students familiar with Web Students agreed that WWW is helpful in finding information (4.2) Students disagreed that information on WWW is not useful (2.1) Students were neutral/agreed that information on WWW is
accurate (3.1) Students disagreed that there is not a lot of detailed or in-depth
information on WWW (2.3)
C R E S S T / U C L A
18
Creating WEA Scales (con’t.)
Searching expertise (α = .71) Number of good searches Quality of keyword searching set Number of redirected searches
In general, students had difficulty searching (consistent with literature) Number of good searches low (M = 2.0) Quality of keyword searching set rather poor
(M = 1.7 on 0-3 scale) Students redirected searches, browsing search
output before selection (M = 2.3)
C R E S S T / U C L A
19
Results
Work explored constructs underlying students’ Web fluency
Students demonstrated their abilities to search, navigate, and find information
Analyses identified important individual measures and we coded these reliably
Composite indicators from factor analysis make sense theoretically
C R E S S T / U C L A
20
Next Steps
Future research will focus on validity of assessment Can we distinguish between expert and novice
Web users with WEA?
Is WEA sensitive to instruction?
Once validity established, use WEA to examine effects of Internet usage
Link assessment and instruction Create guidelines for teaching the Web effectively
C R E S S T / U C L A
21
For More Information
Visit our Web site at: http://www.cse.ucla.edu/CRESST/pages/aera00.htm
Available: Overheads of this presentation
Full paper
And much, much more...