Upload
opensource-connections
View
113
Download
0
Tags:
Embed Size (px)
DESCRIPTION
A refactoring of my earlier presentation targeted more towards Solr, and less on testing. I'm basing my presentation for Lucene EuroCon on this. Would love some more feedback!
Citation preview
WHY AM I QUALIFIED TO BE UP HERE?
• Principal of OpenSource Connections - Solr Search Consultancy
•Member of Apache Software Foundation
• Presenter at conferences (OSCON, ApacheCON, UberConf, ExpoQA, STPcon 2009/2011!)
• Fascinated by the art of software development
2
AUTHOR
3.X Coming
Soon
3
FATHER
4
AGENDA
Why Test Search?
Two General Approaches
Test Tools for Solr
Wrap Up
5
REDUCE HUMAN ERROR
6
photo credit to Jay Grieves @ www.bigswingingdeveloper.com
7
TWO APPROACHES TO TEST
Humans
Automated Tests
8
HUMAN TESTING
• Better deals with ambiguity
9
RELEVANCY IS LIKE PORNOGRAPHY....
10
I KNOW IT WHEN I SEE IT!
http://en.wikipedia.org/wiki/Les_Amants
11
AUTOMATED TESTING12
SWEET SPOT
• Repeat Testing for Relevancy Tuning
• Performance Testing: hitting SLA?
• Regression Testing (Typical use case for automated testing)
13
“ONE TRUE ANSWER”
• Absolute Truth / Matrix / Grid / TREC / Relevancy Assertions
• The correct answers for each search are known ahead of time
• Humans judges often decide these correct answers, stored as Relevancy Assertions
• Can be labor intensive to setup
• A “Numerical Grade” is produced for comparision
14
SO WHAT IS A JUDGEMENT LIST?
15
16
COMPARABLE RESULTSOriginal AjaxSolr WebSolr
gata 4HIF 1 alpha
HIF-1 alpha
HIF1 alpha
HIF1AiNOSIRE-1 alpha
T7 TagCollagen I
Collagen 1
Collagen-1
40.0 80.0 80.090.0 100.0 30
100.0 100.0 30
70.0 100.0 100.0
100.0 100.0 100.060.0 60.0 60.030 90.0 90.0
0.0 100.0 90.050.0 90.0 90.0
40.0 90.0 40.0
10.0 50.0 40.0
17
WRAP UP
18
User Interface
Search Engine
Informa3on
Search Stack
19
20
21
A HYBRID APPROACH?
Humans
Automated Tests
22
A HYBRID APPROACH?
Humans
Automated Tests
Tool SupportedTesting
Judgements23
James Bach,satisfice.com
TESTING IS A HUMAN ACTIVITY
24
25
• twitter: dep4b
• speakerrate: http://www.speakerrate.com/epugh/
• email: [email protected]
THANK YOU!
25