58

Randomized Continuous Testing: Solr & Lucene Use Case

Embed Size (px)

DESCRIPTION

See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011 We have been taught that unit tests should be repeatable and most people (including the author) for a long time considered this an equivalent to "static", single-path execution. Solr and Lucene employ an interesting JUnit runner strategy where tests are randomized -- run with various data, various implementation of allowed interfaces, various configurations. The number of combinations makes running them all impossible, but execution randomization proves very successful at pinpointing implementation and regression bugs. This talk will provide an overview of this approach and practical considerations on when and how to port them to your own projects. Everything that stems from Lucene/Solr is not directly connected to search/ document retrieval and can still be useful and reused in other projects. This session is probably best suited to developers/ CTOs.

Citation preview

Page 1: Randomized Continuous Testing: Solr & Lucene Use Case
Page 2: Randomized Continuous Testing: Solr & Lucene Use Case
Page 3: Randomized Continuous Testing: Solr & Lucene Use Case
Page 4: Randomized Continuous Testing: Solr & Lucene Use Case
Page 5: Randomized Continuous Testing: Solr & Lucene Use Case

!

Page 6: Randomized Continuous Testing: Solr & Lucene Use Case
Page 7: Randomized Continuous Testing: Solr & Lucene Use Case

!!!!!

6WULQJ >@ ZRUGV ^�%DUFHORQD�� �EHDFK�� � �VXQ�� � �RUDFOH� `�SXEOLF YRLG UXQ0HWKRG�5DQGRP U� ^

IRU �LQW L �� L � ���� L��� ^LQW LQGH[

0DWK�DEV�U�QH[W,QW��� � ZRUGV�OHQJWK�6\VWHP�RXW�SULQWOQ�ZRUGV>LQGH[@��

``

#7HVWSXEOLF YRLG WHVW%DUFHORQD:RUGV�� ^UXQ0HWKRG�QHZ 5DQGRP����

`

Page 8: Randomized Continuous Testing: Solr & Lucene Use Case
Page 9: Randomized Continuous Testing: Solr & Lucene Use Case

#7HVW#6HHG�����D��E�� �� JHW XQOXFN\���SXEOLF YRLG WHVW%DUFHORQD��� ^

UXQ0HWKRG�5DQGRPL]HG&RQWH[W�FXUUHQW��

�JHW5DQGRP����`

Page 10: Randomized Continuous Testing: Solr & Lucene Use Case

!!!!!!!!!!!!

� )URP� /XFHQHV B7HVW8WLO� 6WDUW DQG HQG DUH %27+ LQFOXVLYH� �

LQW QH[W,QW�5DQGRP U� LQW VWDUW� LQW HQG� ^UHWXUQ VWDUW � U�QH[W,QW�HQG � VWDUW � ���

`

QH[W,QW�U� �� ,QWHJHU�0$;B9$/8(�

Page 11: Randomized Continuous Testing: Solr & Lucene Use Case

!!!!!!!!!!!!

� )URP� /XFHQHV B7HVW8WLO� 6WDUW DQG HQG DUH %27+ LQFOXVLYH� �

LQW QH[W,QW�5DQGRP U� LQW VWDUW� LQW HQG� ^UHWXUQ VWDUW � U�QH[W,QW�HQG � VWDUW � ���

`

QH[W,QW�U� �� ,QWHJHU�0$;B9$/8(�

Page 12: Randomized Continuous Testing: Solr & Lucene Use Case
Page 13: Randomized Continuous Testing: Solr & Lucene Use Case
Page 14: Randomized Continuous Testing: Solr & Lucene Use Case

!!!!!!!!

!

Page 15: Randomized Continuous Testing: Solr & Lucene Use Case

ILQDO LQW LWHUV DW/HDVW������IRU �LQW LWHU �� LWHU � LWHUV� LWHU��� ^�� ���

`

LI �UDQGRP�QH[W%RROHDQ��� ^WHUP B7HVW8WLO�UDQGRP5HDOLVWLF8QLFRGH6WULQJ�UDQGRP��

` HOVH ^�� ZH ZDQW WR PL[ LQ OLPLWHG�DOSKDEHW V\PEROV VR�� ZH JHW PRUH VKDULQJ RI WKH QRGHV JLYHQ KRZ IHZ�� WHUPV ZH DUH WHVWLQJ���WHUP VLPSOH5DQGRP6WULQJ�UDQGRP��

`

Page 16: Randomized Continuous Testing: Solr & Lucene Use Case

SXEOLF VWDWLF 6WULQJ UDQGRP'LUHFWRU\�5DQGRP UDQGRP� ^LI �UDUHO\�UDQGRP�� ^UHWXUQ &25(B',5(&725,(6>UDQGRP�QH[W,QW�&25(B',5(&725,(6�OHQJWK�@�

` HOVH ^UHWXUQ �5$0'LUHFWRU\��

``

Page 17: Randomized Continuous Testing: Solr & Lucene Use Case

ORFDOH 7(67B/2&$/(�HTXDOV��UDQGRP�� " UDQGRP/RFDOH�UDQGRP� � ���/RFDOH�VHW'HIDXOW�ORFDOH��WLPH=RQH 7(67B7,0(=21(�HTXDOV��UDQGRP�� " UDQGRP7LPH=RQH�UDQGRP� � ���7LPH=RQH�VHW'HIDXOW�WLPH=RQH��

Page 18: Randomized Continuous Testing: Solr & Lucene Use Case
Page 19: Randomized Continuous Testing: Solr & Lucene Use Case
Page 20: Randomized Continuous Testing: Solr & Lucene Use Case
Page 21: Randomized Continuous Testing: Solr & Lucene Use Case

ORQJ VHHG 6\VWHP�FXUUHQW7LPH0LOOLV���6\VWHP�RXW�SULQWOQ�VHHG��5DQGRP UQG QHZ 5DQGRP�VHHG��SDVV7R7HVWV�UQG��

�� #5XQ:LWK�/XFHQH7HVW&DVH5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV /XFHQH7HVW&DVH

�� #5XQ:LWK�5DQGRPL]HG5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV 5DQGRPL]HG7HVW

Page 22: Randomized Continuous Testing: Solr & Lucene Use Case

ORQJ VHHG 6\VWHP�FXUUHQW7LPH0LOOLV���6\VWHP�RXW�SULQWOQ�VHHG��5DQGRP UQG QHZ 5DQGRP�VHHG��SDVV7R7HVWV�UQG��

�� #5XQ:LWK�/XFHQH7HVW&DVH5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV /XFHQH7HVW&DVH

�� #5XQ:LWK�5DQGRPL]HG5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV 5DQGRPL]HG7HVW

Page 23: Randomized Continuous Testing: Solr & Lucene Use Case

ORQJ VHHG 6\VWHP�FXUUHQW7LPH0LOOLV���6\VWHP�RXW�SULQWOQ�VHHG��5DQGRP UQG QHZ 5DQGRP�VHHG��SDVV7R7HVWV�UQG��

�� #5XQ:LWK�/XFHQH7HVW&DVH5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV /XFHQH7HVW&DVH

�� #5XQ:LWK�5DQGRPL]HG5XQQHU�FODVV`SXEOLF FODVV 0\5DQGRPL]HG7HVW

H[WHQGV 5DQGRPL]HG7HVW

Page 24: Randomized Continuous Testing: Solr & Lucene Use Case

assign master seed-Dtests.seed=xxx:...

@Seed(class)mhash(timeMillis)

suite validation

test candidates (+iters)assign method seeds

-Dtests.seed=xxx:yyy@Seed(method)

derive(master, name)

shuffle execution order

@BeforeClass @Before @Test @After @AfterClass

shuffle execution order

Page 25: Randomized Continuous Testing: Solr & Lucene Use Case

SXEOLF FODVV 7HVW&ODVV� H[WHQGV 5DQGRPL]HG7HVW ^#%HIRUH&ODVV SULYDWH VWDWLF YRLG EHIRUH&ODVV�� ^ SULQWOQ��FODVV �� EHIRUH&ODVV��� `#%HIRUH SULYDWH YRLG EHIRUH�� ^ SULQWOQ�� FODVV �� EHIRUH��� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#$IWHU SULYDWH YRLG DIWHU�� ^ SULQWOQ�� FODVV �� DIWHU��� `#$IWHU&ODVV SULYDWH VWDWLF YRLG DIWHU&ODVV�� ^ SULQWOQ��FODVV �� DIWHU&ODVV��� `

`

SXEOLF FODVV 7HVW&ODVV� H[WHQGV 7HVW&ODVV� ^#%HIRUH&ODVV SULYDWH VWDWLF YRLG EHIRUH&ODVV�� ^ SULQWOQ��FODVV �� EHIRUH&ODVV��� `#%HIRUH SULYDWH YRLG EHIRUH�� ^ SULQWOQ�� FODVV �� EHIRUH��� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#7HVW SXEOLF YRLG WHVW�B��� ^ SULQWOQ�� FODVV �� WHVW���� `#$IWHU SULYDWH YRLG DIWHU�� ^ SULQWOQ�� FODVV �� DIWHU��� `#$IWHU&ODVV SULYDWH VWDWLF YRLG DIWHU&ODVV�� ^ SULQWOQ��FODVV �� DIWHU&ODVV��� `

`

Page 26: Randomized Continuous Testing: Solr & Lucene Use Case

FODVV �� EHIRUH&ODVVFODVV �� EHIRUH&ODVVFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHU

FODVV �� DIWHU&ODVVFODVV �� DIWHU&ODVV

FODVV �� EHIRUH&ODVVFODVV �� EHIRUH&ODVVFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHU

FODVV �� DIWHU&ODVVFODVV �� DIWHU&ODVV

FODVV �� EHIRUH&ODVVFODVV �� EHIRUH&ODVV

FODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHUFODVV �� EHIRUHFODVV �� EHIRUHFODVV �� WHVW�

FODVV �� DIWHUFODVV �� DIWHU

FODVV �� DIWHU&ODVVFODVV �� DIWHU&ODVV

Page 27: Randomized Continuous Testing: Solr & Lucene Use Case
Page 28: Randomized Continuous Testing: Solr & Lucene Use Case

5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���

FW[�JHW7DUJHW&ODVV��FW[�JHW5DQGRP�� �� ! QR VHW6HHG���FW[�JHW5DQGRPQHVV���JHW6HHG��FW[�LV1LJKWO\�����

#1LJKWO\ #7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW1LJKWO\�� ^� VQRR]H �`

#7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW$OZD\V�� ^

DVVHUW7UXH�UDQGRP%RROHDQ����`

Page 29: Randomized Continuous Testing: Solr & Lucene Use Case

5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���

FW[�JHW7DUJHW&ODVV��FW[�JHW5DQGRP�� �� ! QR VHW6HHG���FW[�JHW5DQGRPQHVV���JHW6HHG��FW[�LV1LJKWO\�����

#1LJKWO\ #7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW1LJKWO\�� ^� VQRR]H �`

#7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW$OZD\V�� ^

DVVHUW7UXH�UDQGRP%RROHDQ����`

Page 30: Randomized Continuous Testing: Solr & Lucene Use Case

5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���

FW[�JHW7DUJHW&ODVV��FW[�JHW5DQGRP�� �� ! QR VHW6HHG���FW[�JHW5DQGRPQHVV���JHW6HHG��FW[�LV1LJKWO\�����

#1LJKWO\ #7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW1LJKWO\�� ^� VQRR]H �`

#7HVW #5HSHDW�LWHUDWLRQV ��SXEOLF YRLG WHVW$OZD\V�� ^

DVVHUW7UXH�UDQGRP%RROHDQ����`

Page 31: Randomized Continuous Testing: Solr & Lucene Use Case
Page 32: Randomized Continuous Testing: Solr & Lucene Use Case
Page 33: Randomized Continuous Testing: Solr & Lucene Use Case

#6HHG��GHDGEHHI��#7HVW #5HSHDW�LWHUDWLRQV ��� XVH&RQVWDQW6HHG WUXH�SXEOLF YRLG WHVW$OZD\V�� ^DVVHUW7UXH�UDQGRP%RROHDQ����

`

�'WHVWV�VHHG ��GHDGEHHI �� VHHG IRUPDW� PDVWHU>�PHWKRG@�'WHVWV�LWHU ���'WHVWV�PHWKRG WHVW$OZD\V

Page 34: Randomized Continuous Testing: Solr & Lucene Use Case

#6HHG��GHDGEHHI��#7HVW #5HSHDW�LWHUDWLRQV ��� XVH&RQVWDQW6HHG WUXH�SXEOLF YRLG WHVW$OZD\V�� ^DVVHUW7UXH�UDQGRP%RROHDQ����

`

�'WHVWV�VHHG ��GHDGEHHI �� VHHG IRUPDW� PDVWHU>�PHWKRG@�'WHVWV�LWHU ���'WHVWV�PHWKRG WHVW$OZD\V

Page 35: Randomized Continuous Testing: Solr & Lucene Use Case
Page 36: Randomized Continuous Testing: Solr & Lucene Use Case

#7HVW #7LPHRXW�PLOOLV �����SXEOLF YRLG LQWHUUXSWDEOH�� WKURZV ([FHSWLRQ ^

7KUHDG�VOHHS���������`

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� $WWHPSWLQJ WR VWRS WKUHDG� 7KUHDG���������������� FXUUHQWO\ DW�

MDYD�ODQJ�7KUHDG�VOHHS�1DWLYH 0HWKRG�UDQGRPL]HGWHVWLQJ�H[DPSOHV�7HVW([DPSOH��LQWHUUXSWDEOH�7HVW([DPSOH��MDYD����VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH��1DWLYH 0HWKRG�VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH�1DWLYH0HWKRG$FFHVVRU,PSO�MDYD����VXQ�UHIOHFW�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�LQYRNH�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�MDYD����MDYD�ODQJ�UHIOHFW�0HWKRG�LQYRNH�0HWKRG�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�LQYRNH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���HYDOXDWH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ:LWK5XOHV�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ6LQJOH7HVW�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���UXQ�5DQGRPL]HG5XQQHU�MDYD�����MDYD�ODQJ�7KUHDG�UXQ�7KUHDG�MDYD�����

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� ,QWHUUXSWHG D UXQDZD\ WKUHDG� 7KUHDG���������������

Page 37: Randomized Continuous Testing: Solr & Lucene Use Case

#7HVW #7LPHRXW�PLOOLV �����SXEOLF YRLG XQLQWHUUXSWDEOH�� WKURZV ([FHSWLRQ ^

ZKLOH �WUXH� � VSLQ ��`

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� $WWHPSWLQJ WR VWRS WKUHDG� 7KUHDG���������������� FXUUHQWO\ DW�

UDQGRPL]HGWHVWLQJ�H[DPSOHV�7HVW([DPSOH��XQLQWHUUXSWDEOH�7HVW([DPSOH��MDYD����VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH��1DWLYH 0HWKRG�VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH�1DWLYH0HWKRG$FFHVVRU,PSO�MDYD����VXQ�UHIOHFW�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�LQYRNH�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�MDYD����MDYD�ODQJ�UHIOHFW�0HWKRG�LQYRNH�0HWKRG�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�LQYRNH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���HYDOXDWH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ:LWK5XOHV�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ6LQJOH7HVW�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���UXQ�5DQGRPL]HG5XQQHU�MDYD�����MDYD�ODQJ�7KUHDG�UXQ�7KUHDG�MDYD�����

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� 'RHV QRW UHVSRQG WR LQWHUUXSW��� WU\LQJ WR VWRS��� 7KUHDG���������������2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� 6WRSSHG D UXQDZD\ WKUHDG� 7KUHDG���������������

Page 38: Randomized Continuous Testing: Solr & Lucene Use Case

#7HVW #7LPHRXW�PLOOLV �����SXEOLF YRLG FKULVWRSKHU/DPEHUW�� WKURZV ([FHSWLRQ ^

ZKLOH �WUXH� ^WU\ ^ 7KUHDG�VOHHS������� ` FDWFK �7KURZDEOH W� ^`

``

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� $WWHPSWLQJ WR VWRS WKUHDG� 7KUHDG���������������� FXUUHQWO\ DW�

MDYD�ODQJ�7KUHDG�VOHHS�1DWLYH 0HWKRG�UDQGRPL]HGWHVWLQJ�H[DPSOHV�7HVW([DPSOH��FKULVWRSKHU/DPEHUW�7HVW([DPSOH��MDYD����VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH��1DWLYH 0HWKRG�VXQ�UHIOHFW�1DWLYH0HWKRG$FFHVVRU,PSO�LQYRNH�1DWLYH0HWKRG$FFHVVRU,PSO�MDYD����VXQ�UHIOHFW�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�LQYRNH�'HOHJDWLQJ0HWKRG$FFHVVRU,PSO�MDYD����MDYD�ODQJ�UHIOHFW�0HWKRG�LQYRNH�0HWKRG�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�LQYRNH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���HYDOXDWH�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ:LWK5XOHV�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�UXQ6LQJOH7HVW�5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU�DFFHVV���5DQGRPL]HG5XQQHU�MDYD�����UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU���UXQ�5DQGRPL]HG5XQQHU�MDYD�����MDYD�ODQJ�7KUHDG�UXQ�7KUHDG�MDYD�����

2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH:$51,1*� 'RHV QRW UHVSRQG WR LQWHUUXSW��� WU\LQJ WR VWRS��� 7KUHDG���������������2FW ��� ���� ������� 30 UDQGRPL]HGWHVWLQJ�5DQGRPL]HG5XQQHU WU\7R7HUPLQDWH6(9(5(� &RXOG QRW LQWHUUXSW RU VWRS WKUHDG� 7KUHDG���������������

Page 39: Randomized Continuous Testing: Solr & Lucene Use Case
Page 40: Randomized Continuous Testing: Solr & Lucene Use Case
Page 41: Randomized Continuous Testing: Solr & Lucene Use Case
Page 42: Randomized Continuous Testing: Solr & Lucene Use Case
Page 43: Randomized Continuous Testing: Solr & Lucene Use Case
Page 44: Randomized Continuous Testing: Solr & Lucene Use Case

6WULQJ UHDG)LOH�)LOH ILOH� &KDUVHW FKDUVHW� ^E\WH>@ E\WHV QHZ E\WH>�LQW� ILOH�OHQJWK��@�'DWD,QSXW6WUHDP LV

QHZ 'DWD,QSXW6WUHDP�QHZ )LOH,QSXW6WUHDP�ILOH���WU\ ^

LV�UHDG)XOO\�E\WHV��` ILQDOO\ ^ LV�FORVH��� `UHWXUQ QHZ 6WULQJ�E\WHV� FKDUVHW��

`

#7HVW #5HSHDW�LWHUDWLRQV ���SXEOLF YRLG WHVW5HDG)LOH�� WKURZV ,2([FHSWLRQ ^)LOH WHPS)LOH QHZ7HPS)LOH���)LOH2XWSXW6WUHDP IRV QHZ )LOH2XWSXW6WUHDP�WHPS)LOH��6WULQJ FRQWHQWV UDQGRP8QLFRGH6WULQJ���IRV�ZULWH�FRQWHQWV�JHW%\WHV�87)����IRV�FORVH���

6WULQJ UHDG%DFN ,28WLOV�UHDG)LOH�WHPS)LOH� 87)���$VVHUW�DVVHUW(TXDOV�FRQWHQWV� UHDG%DFN��

`

Page 45: Randomized Continuous Testing: Solr & Lucene Use Case

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

Page 46: Randomized Continuous Testing: Solr & Lucene Use Case

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

Page 47: Randomized Continuous Testing: Solr & Lucene Use Case

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� 6LPXODWH D UDQGRP ,�2 VORZGRZQ� �EHIRUH�� � UHDG6WUHDP�� WHVWLQJ7KUHDG�� ^5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���5DQGRP U FW[�JHW5DQGRP����� ��� ,�2 HUURUV UDWH�LI �U�QH[W,QW����� � ��� ^WU\ ^7KUHDG�VOHHS�UDQGRP,QW%HWZHHQ�U� ���� �������

` FDWFK �,QWHUUXSWHG([FHSWLRQ H� ^� LJQRUH �``

`

Page 48: Randomized Continuous Testing: Solr & Lucene Use Case

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� 6LPXODWH D UDQGRP ,�2 VORZGRZQ� �EHIRUH�� � UHDG6WUHDP�� WHVWLQJ7KUHDG�� ^5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW���5DQGRP U FW[�JHW5DQGRP����� ��� ,�2 HUURUV UDWH�LI �U�QH[W,QW����� � ��� ^WU\ ^7KUHDG�VOHHS�UDQGRP,QW%HWZHHQ�U� ���� �������

` FDWFK �,QWHUUXSWHG([FHSWLRQ H� ^� LJQRUH �``

`

Page 49: Randomized Continuous Testing: Solr & Lucene Use Case

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� 6LPXODWH D UDUH ,�2 SUREOHP ZKHQ RSHQLQJ ILOHV� �2EMHFW DURXQG�� � RSHQ)LOH�� WHVWLQJ7KUHDG�� ^5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW����� ��� ,�2 HUURUV UDWH�LI �FW[�JHW5DQGRP���QH[W,QW����� � ��� ^5HWKURZ�UHWKURZ�QHZ ,2([FHSWLRQ�

�6LPXODWHG ,�2 H[FHSWLRQ RQ ILOH RSHQLQJ�����`UHWXUQ SURFHHG���

`

Page 50: Randomized Continuous Testing: Solr & Lucene Use Case

� )LOH RSHQLQJ SRLQWFXWV �VLPSOLVWLF�� �SRLQWFXW RSHQ)LOH���

FDOO�MDYD�LR�)LOH,QSXW6WUHDP�QHZ����� __FDOO�MDYD�LR�5DQGRP$FFHVV)LOH�QHZ������

� $Q\ UHDGLQJ IURP DQ\ VWUHDP� �SRLQWFXW UHDG6WUHDP���

FDOO� MDYD�LR�,QSXW6WUHDP��UHDG ������

� $OO WHVW PHWKRGV� �SRLQWFXW WHVW0HWKRGV��� H[HFXWLRQ�#7HVW ����

� ([HFXWLRQ XQGHU D WHVW PHWKRG� �SRLQWFXW WHVWLQJ7KUHDG��� FIORZEHORZ�WHVW0HWKRGV����

� 6LPXODWH D UDUH ,�2 SUREOHP ZKHQ RSHQLQJ ILOHV� �2EMHFW DURXQG�� � RSHQ)LOH�� WHVWLQJ7KUHDG�� ^5DQGRPL]HG&RQWH[W FW[ 5DQGRPL]HG&RQWH[W�FXUUHQW����� ��� ,�2 HUURUV UDWH�LI �FW[�JHW5DQGRP���QH[W,QW����� � ��� ^5HWKURZ�UHWKURZ�QHZ ,2([FHSWLRQ�

�6LPXODWHG ,�2 H[FHSWLRQ RQ ILOH RSHQLQJ�����`UHWXUQ SURFHHG���

`

Page 51: Randomized Continuous Testing: Solr & Lucene Use Case
Page 52: Randomized Continuous Testing: Solr & Lucene Use Case
Page 53: Randomized Continuous Testing: Solr & Lucene Use Case

SRLQWFXW QHZ7HPS'LU���FDOO�VWDWLF )LOH 5DQGRPL]HG7HVW�QHZ7HPS'LU����

� 5HPHPEHU ZKHQ D WHPSRUDU\ IROGHU LV UHTXHVWHG� �)LOH DURXQG�� � QHZ7HPS'LU�� WHVWLQJ7KUHDG�� ^)LOH GLU SURFHHG����� 72'2� 5HPHPEHU WKH FXUUHQW WKUHDGV VWDFN�� DQG WKH IDFW LW DOORFDWHG GLUUHWXUQ GLU�

`

� $IWHU D WHVW PHWKRG H[LWV FKHFN DVVHUWLRQV� �DIWHU�� � WHVW0HWKRGV�� ^

�� 72'2� IRU WKH H[LWLQJ WKUHDG� HQVXUH DOO�� DOORFDWHG GLUV FDQ EH UHPRYHG �DUH QRW ORFNHG���� ,I WKH\ DUH ORFNHG� IDLO WKH WHVW FDVH DQG UHSRUW�� WKH GLUV DOORFDWLRQ VWDFN�`

`

Page 54: Randomized Continuous Testing: Solr & Lucene Use Case

SRLQWFXW QHZ7HPS'LU���FDOO�VWDWLF )LOH 5DQGRPL]HG7HVW�QHZ7HPS'LU����

� 5HPHPEHU ZKHQ D WHPSRUDU\ IROGHU LV UHTXHVWHG� �)LOH DURXQG�� � QHZ7HPS'LU�� WHVWLQJ7KUHDG�� ^)LOH GLU SURFHHG����� 72'2� 5HPHPEHU WKH FXUUHQW WKUHDGV VWDFN�� DQG WKH IDFW LW DOORFDWHG GLUUHWXUQ GLU�

`

� $IWHU D WHVW PHWKRG H[LWV FKHFN DVVHUWLRQV� �DIWHU�� � WHVW0HWKRGV�� ^

�� 72'2� IRU WKH H[LWLQJ WKUHDG� HQVXUH DOO�� DOORFDWHG GLUV FDQ EH UHPRYHG �DUH QRW ORFNHG���� ,I WKH\ DUH ORFNHG� IDLO WKH WHVW FDVH DQG UHSRUW�� WKH GLUV DOORFDWLRQ VWDFN�`

`

Page 55: Randomized Continuous Testing: Solr & Lucene Use Case
Page 56: Randomized Continuous Testing: Solr & Lucene Use Case
Page 57: Randomized Continuous Testing: Solr & Lucene Use Case
Page 58: Randomized Continuous Testing: Solr & Lucene Use Case

GDZLG�ZHLVV#FDUURWVHDUFK�FRP