13
BT12 Concurrent Session 11/12/15 3:00pm “Agile Automation Strategies and Frameworks” Presented by: Max Saperstone Coveros Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 · 904-278-0524 · [email protected] · www.techwell.com

Agile Automation Strategies and Frameworks

Embed Size (px)

Citation preview

Page 1: Agile Automation Strategies and Frameworks

BT12 ConcurrentSession11/12/153:00pm

“Agile Automation Strategies and Frameworks”

Presented by:

Max Saperstone

Coveros

Broughttoyouby:

340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com

Page 2: Agile Automation Strategies and Frameworks

Max Saperstone Coveros

For almost a decade, Max Saperstone has been a test engineer focusing on test automation and the continuous integration/continuous delivery process. Max specializes in open source tools—Selenium, JMeter, AutoIT, Cucumber, and Chef. He has led several testing automation efforts, including developing an automated suite focused on web-based software to operate over several applications. Max also headed a major project developing an automated testing structure to run Cucumber tests over multiple test interfaces and environments, while developing a system to keep test data “ageless.” He is currently developing a new testing architecture for SecureCI to allow testing of multiple interfaces, custom reporting, and minimal test upkeep.

Page 3: Agile Automation Strategies and Frameworks

��&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�6WUDWHJLHV�DQG�)UDPHZRUNV

:KDW�6KRXOG�<RXU�7HDP�'R"

0D[�6DSHUVWRQH

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

0D[�6DSHUVWRQH

0D[�6DSHUVWRQH�KDV�EHHQ�ZRUNLQJ�DV�D�6RIWZDUH�DQG�7HVW�(QJLQHHU�IRU�DOPRVW�D�GHFDGH��ZLWK�D�IRFXV�RQ�7HVW�$XWRPDWLRQ�DQG�WKH�&,�&'�SURFHVV��+H�VSHFLDOL]HV�LQ�RSHQ�VRXUFH�WRROV��LQFOXGLQJ�WKH�6HOHQLXP�7RRO�6XLWH��-0HWHU��$XWR,7��&XFXPEHU��DQG�&KHI��0D[�KDV�OHDG�VHYHUDO�WHVWLQJ�DXWRPDWLRQ�HIIRUWV��LQFOXGLQJ�GHYHORSLQJ�DQ�DXWRPDWHG�VXLWH�IRFXVLQJ�RQ�ZHE�EDVHG�VRIWZDUH�WR�RSHUDWH�RYHU�VHYHUDO�DSSOLFDWLRQV�IRU�.URQRV�)HGHUDO��+H�DOVR�KHDGHG�D�SURMHFW�ZLWK�'HOWD�'HQWDO��GHYHORSLQJ�DQ�DXWRPDWHG�WHVWLQJ�VWUXFWXUH�WR�UXQ�&XFXPEHU�WHVWV�RYHU�PXOWLSOH�WHVW�LQWHUIDFHV�DQG�HQYLURQPHQWV��ZKLOH�DOVR�GHYHORSLQJ�D�V\VWHP�WR�NHHS�WHVW�GDWD�DJHOHVV��+H�UHFHQWO\�UHOHDVHG�D�QHZ�WHVWLQJ�DUFKLWHFWXUH�IRU�6HFXUH&,��WR�DOORZ�WHVWLQJ�RI�PXOWLSOH�LQWHUIDFHV��FXVWRP�UHSRUWLQJ��DQG�PLQLPDO�WHVW�XSNHHS��+H�DOVR�LV�SURYLGLQJ�VXSSRUW�WR�WKH�&XFXPEHU�FRPPXQLW\�E\�EXLOGLQJ�VRIWZDUH�WR�PDNH�%''�WHVW�VWHSV�EXLOGDEOH��DFFHVVLEOH��DQG�VHDUFKDEOH�DFURVV�DQ�HQWLUH�WHVWLQJ�WHDP��+H�LV�FXUUHQWO\�HQJDJHG�LQ�'HY2SV�ZRUN��ZRUNLQJ�WR�FUHDWH�IXOO\�DXWRPDWHG�GHOLYHU\�V\VWHP�XVLQJ�RSHQ�VRXUFH�WRROV�LQFOXGLQJ�-HQNLQV��6RQDU4XEH��DQG�1H[XV�

Page 4: Agile Automation Strategies and Frameworks

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

$ERXW�&RYHURV

� &RYHURV�KHOSV�RUJDQL]DWLRQV�DFFHOHUDWH�WKH�GHOLYHU\�RI�VHFXUH��UHOLDEOH�VRIWZDUH

� 2XU�FRQVXOWLQJ�VHUYLFHV�± $JLOH�VRIWZDUH�GHYHORSPHQW± $SSOLFDWLRQ�VHFXULW\± 6RIWZDUH�TXDOLW\�DVVXUDQFH± 6RIWZDUH�SURFHVV�LPSURYHPHQW

� 2XU�NH\�PDUNHWV�± )LQDQFLDO�VHUYLFHV± +HDOWKFDUH�± 'HIHQVH± &ULWLFDO�,QIUDVWUXFWXUH

'HYHORSPHQW�&DSDELOLWLHV

��&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�3ODQQLQJ

Page 5: Agile Automation Strategies and Frameworks

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�3ODQQLQJ

Ɣ 60$57ż 6SHFLILF��:KDW�WR�DFFRPSOLVK��ZK\��ZKR��ZKHUH�±�VSHFLILF�UHTV�ż 0HDVXUDEOH��+RZ�PXFK�PDQ\�±�4XDQWLILDEOH�ZD\�WR�GHWHUPLQH�

FRPSOHWHQHVV�ż $FKLHYDEOH��DQVZHUV�TXHVWLRQ�RI�³+RZ�ZLOO�WKLV�EH�DFFRPSOLVKHG"´�ż 5HOHYDQW��9DOXHG�:RUWKZKLOH��FRQVLGHU�WLPLQJ�SULRULWLHV�ż 7LPH�%RXQG��WDUJHW�FRPSOHWLRQ�

Ɣ 6RPH�W\SLFDO�REMHFWLYHV�ż 7HVWLQJ�&RQVLVWHQF\�5HSHDWDELOLW\ż 8QDWWHQGHG�WHVW�H[HFXWLRQż 5HJUHVVLRQ�GHIHFW�LGHQWLILFDWLRQż )UHTXHQW�7HVW�([HFXWLRQż 6RIWZDUH�4XDOLW\�,PSURYHPHQW

7HVW�$XWRPDWLRQ�2EMHFWLYHV

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�3ODQQLQJ

Ɣ 8VDELOLW\Ɣ $FFXUDF\Ɣ ,QWHJULW\Ɣ ,QWHURSHUDELOLW\�DQG�&RPSDWLELOLW\Ɣ 0DLQWDLQDELOLW\Ɣ $YDLODELOLW\Ɣ 3HUIRUPDQFH

$UHDV�WR�FRQVLGHU�IRU�7HVW�2EMHFWLYHV

Page 6: Agile Automation Strategies and Frameworks

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�3ODQQLQJ

Ɣ :KDW�LV�JRLQJ�WR�EH�FRQVLGHUHG�LQ�VFRSH�IRU�WHVWLQJ"Ɣ :KDW�ZLOO�EH�RXW�RI�VFRSH"Ɣ $W�ZKLFK�OHYHO�RI�WHVWLQJ�ZLOO�ZH�LQFUHDVH�RU�GHFUHDVH�RXU�

VFRSH�WR�FRYHU�WKHVH"Ɣ :KDW�LV�LQ�VFRSH�IRU�DXWRPDWLRQ�YV

PDQXDO�WHVWLQJ

7HVW�$XWRPDWLRQ�6FRSH

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

Ɣ 3ULPDULO\�RQO\�GRQH�IRU�QRQ�IXQFWLRQDO�UHTXLUHPHQWV��VPRNH�WHVWV��EXVLQHVV�SURFHVV

Ɣ $FFHSWDQFH�WHVW�GULYHQ�GHYHORSPHQW�6WRU\�WHVWLQJ

Ɣ 7HVW�GULYHQ�GHYHORSPHQW�8QLW�WHVWLQJ

7HVW�$XWRPDWLRQ�3ODQQLQJ

7HVW�7\SHV�WR�,QFOXGH

Page 7: Agile Automation Strategies and Frameworks

��&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�)UDPHZRUN�6HOHFWLRQ

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�)UDPHZRUN�6HOHFWLRQ

Ɣ )UDPHZRUNV�SURYLGH�WHVW�LQIUDVWUXFWXUH�IRU�DXWRPDWLQJ�YDULRXV�W\SHV�DQG�OHYHOV�RI�WHVWV�ż &DSDELOLWLHV�IRU�FRPPRQ�DXWRPDWLRQ�UHTXLUHPHQWVż &RPSOH[LW\�$EVWUDFWLRQż ([WHQG�&RYHUDJHż 5HGXFH�0DLQWHQDQFH

Ɣ )UDPHZRUN�)HDWXUHVż 7HVW�'ULYHUż 2EMHFW�5HSRVLWRU\�0DQDJHPHQWż 'DWD�0DQDJHPHQWż 5HFRYHU\�0HFKDQLVPV

:KDW�LV�D�)UDPHZRUN"

Page 8: Agile Automation Strategies and Frameworks

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�)UDPHZRUN�6HOHFWLRQ

Ɣ 5HFRUG�3OD\EDFN��/LQHDU�Ɣ 0RGXODU���6WUXFWXUHGƔ 'DWD�'ULYHQƔ .H\ZRUG�'ULYHQƔ $FWLRQ�%DVHGƔ +\EULGV

$XWRPDWLRQ�)UDPHZRUN�7\SHV

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�)UDPHZRUN�6HOHFWLRQ

Ɣ 'HSHQGHQF\�UHGXFWLRQƔ &RYHUDJHƔ ,QFUHPHQWDO�FRVW�IRU�)XWXUH�(QKDQFHPHQWVƔ &RVW�LV�(VWLPDEOH

%HQHILWV�RI�D�)UDPHZRUN

Page 9: Agile Automation Strategies and Frameworks

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�)UDPHZRUN�6HOHFWLRQ

Ɣ 7KHUH�LV�QR�µ6LOYHU�%XOOHW¶Ɣ 7RRO�(YDOXDWLRQ

ż 0RWLYDWLRQż 3URMHFW�7HFKQLFDO�FRQVWUDLQWV

Ŷ 26�/DQJXDJHŶ 'HYHORSPHQW�,'(Ŷ 3URMHFW�$UFKLWHFWXUH

ż 7HDP�VNLOOVHWV�H[SHUWLVHż %XGJHWż %HVW�WRRO�ILW�VHOHFWLRQ

7HVW�$XWRPDWLRQ�7RRO�6HOHFWLRQ

��&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�6WUDWHJ\

Page 10: Agile Automation Strategies and Frameworks

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�6WUDWHJ\

/LPLWHG�7LPH���,QLWLDO�(IIRUW�UHTXLUHG� �0R6&R:�PHWKRGƔ :KDW�0867�EH�$XWRPDWHG"Ɣ :KDW�6+28/'�EH�$XWRPDWHG"Ɣ :KDW�&28/'�EH�$XWRPDWHG"Ɣ :KDW�:21¶7�EH�$XWRPDWHG"

6HOHFWLQJ�7HVWV�IRU�$XWRPDWLRQ

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�6WUDWHJ\

Ɣ 9DOXH�YV�(IIRUW�HYDOXDWLRQƔ )UHTXHQF\�RI�8VH

ż 5HSHWLWLYH�WHVWV�WKDW�UXQ�IRU�PXOWLSOH�EXLOGV�ż 7HGLRXV�0DQXDO�7DVNV

Ɣ )HDWXUHV�WKDW�KLVWRULFDOO\�SUREOHPDWLF�SURQH�WR�GHIHFWVƔ 7HVWV�WKDW�DUH�KLJKO\�VXEMHFW�WR�KXPDQ�HUURUƔ 7HVWV�WKDW�UHTXLUH�PXOWLSOH�GDWD�VHWVƔ 7HVWV�WKDW�UXQ�RQ�VHYHUDO�GLIIHUHQW�+:�6:�SODWIRUPV�DQG�

FRQILJXUDWLRQVƔ 7HVWV�WKDW�WDNH�D�ORW�RI�HIIRUW�DQG�WLPH�ZKHQ�GRLQJ�PDQXDO�

WHVWLQJ

'HWHUPLQLQJ�ZKDW�WR�$XWRPDWH

Page 11: Agile Automation Strategies and Frameworks

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�6WUDWHJ\

Ɣ 6KRXOG�GDWD�VWDUW�HPSW\��DQG�EH�FOHDQHG�HDFK�WLPH"Ɣ :KDW�GDWD�LV�QHHGHG�LQ�4$�IRU�DXWRPDWHG�WHVWLQJ"�:KDW�

DERXW�PDQXDO�WHVWLQJ"Ɣ +RZ�FDQ�ZH�NHHS�4$�LQ�D�VWDWH�ZLWK�NQRZQ�WHVW�GDWD"Ɣ +RZ�GR�ZH�KDQGOH�PXOWLSOH�XVHUV�DXWRPDWHG�WHVWV�

SRWHQWLDOO\�FKDQJLQJ�DQG�QHHGHG�WKH�VDPH�GDWD"Ɣ :KHUH�LV�LW�DFFHSWDEOH�WR�XVH�SURGXFWLRQ�GDWD"Ɣ 6KRXOG�SURGXFWLRQ�GDWD�EH�VDQLWL]HG�EHIRUH�EHLQJ�XVHG"Ɣ 6KRXOG�SURGXFWLRQ�GDWD�EH�DQDO\]HG�EHIRUH�EHLQJ�XVHG"

'DWD�0DQDJHPHQW

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�6WUDWHJ\

Ɣ )UDPHZRUNV�VKRXOG�LQFOXGH�WDNH�VFUHHQVKRWV�PHWKRGV�ZKLFK�FDQ�EH�WDNHQ�RQ�DQ\�IDLOHG�VWHSVż 7KHVH�FDQ�DOVR�EH�LQFOXGHG�IRU�DGGLWLRQDO�LQIRUPDWLRQ

Ɣ $XWRPDWHG�IUDPHZRUNV�VKRXOG�SURYLGH�UHDGDEOH�UHSRUWV��DQG�EH�XVHG�IRU�GRFXPHQWDWLRQ

Ɣ 5HVXOWV�IURP�IUDPHZRUNV�VKRXOG�EH�DXWRPDWHG�WR�SURYLGH�WUDFHDELOLW\�LQWR�UHTXLUHPHQWVż 0XOWLSOH�WRROV�SURYLGH�KRRNV�LQWR�WHVW�PDQDJHPHQW�V\VWHPVż 3XEOLF�$3,V�FDQ�DOORZ�DXWRPDWHG�XSGDWHV�IRU�WRROLQJ

7UDFHDELOLW\�WKURXJK�$XWRPDWHG�7HVWLQJ

Page 12: Agile Automation Strategies and Frameworks

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�6WUDWHJ\

Ɣ <RXU�DXWRPDWHG�WHVWV�VKRXOG�EH�WUHDWHG�DV�FRGHƔ :KHQ�FUHDWHG�VPDUWO\��VWHSV�DUH�VLPSOH�WR�ILQG��XSGDWH��

DQG�UHXVHƔ &UHDWH�D�VHSDUDWH�FODVV�IRU�HDFK�DUHD�RI�IXQFWLRQDOLW\Ɣ ,QVWDQWLDWH�LPSRUWDQW�FODVVHV�RU�REMHFWV�LQ�\RXU�EDVH�FODVV

%DVH�)XQFWLRQDOLW\

+HOS�)XQFWLRQDOLW\/RJLQ�)XQFWLRQDOLW\ /LEUDU\�)XQFWLRQDOLW\

%HVW�3UDFWLFHV

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�6WUDWHJ\

Ɣ <RX�FDQW�WHVW�DOO�SRVVLEOH�FRPELQDWLRQV�RI�SDUDPHWHUVƔ 8VLQJ�SDUDPHWHUL]HG�LQSXWV�WR�D�PHWKRG�DOORZV�IRU�D�

VLPSOH�H[SDQVLRQ�RI�WHVW�VXLWHVż 7KLV�DOORZV�RQH�WHVW�WR�UXQ�ZLWK�PXOWLSOH�GDWD�VHWVż *HQHUDOO\��WKHVH�FDQ�UXQ�LQ�SDUDOOHO

Ɣ 0XOWLSOH�H[LVWLQJ�IUDPHZRUNV�VXSSRUW�SDUDPHWHUL]HG�WHVWLQJ

Ɣ 7DNH�\RXU�VPRNH�DQG�TXLFNHUH[HFXWLQJ�WHVWV�DQG�H[SDQG�RQ�WKHP�XVLQJ�SDUDPHWHUV

3DUDPHWHUL]HG�7HVWLQJ

Page 13: Agile Automation Strategies and Frameworks

���&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

7HVW�$XWRPDWLRQ�)UDPHZRUNV

Ɣ $EERWƔ $QWHDWHUƔ &DQRR�:HE7HVWƔ &KRUXVƔ &RQFRUGLRQƔ &URVVFKHFNƔ 'HMD*QXƔ 'RJWDLOƔ (026Ɣ ,WL1Ɣ ,WV1DW

Ɣ -%HKDYHƔ 0DYHU\[Ɣ 0R]8QLWƔ 3KRURQL[�7HVW�6XLWHƔ 4$OLEHUƔ 5DVWDƔ 5RERWƔ 5RERWLXPƔ 7HVW1*Ɣ 6HOHQGURLGƔ :$$7

:KDW�7HVWLQJ�)UDPHZRUN�0LJKW�%H�5LJKW�)RU�0H"

��&RS\ULJKW������&RYHURV��,QF��$OO�ULJKWV�UHVHUYHG�

4XHVWLRQV