Automated UI test on mobile - with Cucumber/Calabash

Mobiltest automation - experiences from the trenches

Let’s Talk Apple, 11. sep, 2014 Niels Frydenholm, ebay Classifieds

eBay Inc. Confidential2

Agenda

• Intro  

• Tools  

• Examples  /  Demo  

• Continuous  Integration  /  Follow  up  

• Challenges  

• Latest  improvements  

• Questions?

eBay Inc. Confidential3

eBay  Classifieds,  Denmark

• DBA,  BilBasen  og  Bilinfo  

• DBA  (a  classifieds  site/app)  has  approx.  1  mill  downloads    

• BilBasen  (cars)  has  approx.  450.000  downloads  

• Both  brands  have  apps  for  iPhone,  iPad  and  Android    

• Traffic  from  mobile  doubled  during  2013  

• Strive  for  rapid  releases

eBay Inc. Confidential4

How  we  are  organised

• Developers • QA

• Product Owner • UX • Other stakeholders

eBay Inc. Confidential5

Tools

Jenkins (CI)

eBay Inc. Confidential6

Cucumber  -­‐  more  than  a  test  tool

eBay Inc. Confidential6

Cucumber  -­‐  more  than  a  test  tool

eBay Inc. Confidential6

Cucumber  -­‐  more  than  a  test  tool

• Behaviour-­‐Driven  Development  

• Adds  “power”  to  Gherkin  syntax  

• A  great  way  to  communicate  and  cooperate  between  “the  business  people”,  developers  and  QA.  

• Written  in  (your)  business/domain  language    

• Easy  to  both  read  and  write

eBay Inc. Confidential6

Cucumber  -­‐  more  than  a  test  tool

• Behaviour-­‐Driven  Development  

• Adds  “power”  to  Gherkin  syntax  

• A  great  way  to  communicate  and  cooperate  between  “the  business  people”,  developers  and  QA.  

• Written  in  (your)  business/domain  language    

• Easy  to  both  read  and  write

Feature: Simple search As a user I want to be able to do simple searches

eBay Inc. Confidential6

Cucumber  -­‐  more  than  a  test  tool

• Behaviour-­‐Driven  Development  

• Adds  “power”  to  Gherkin  syntax  

• A  great  way  to  communicate  and  cooperate  between  “the  business  people”,  developers  and  QA.  

• Written  in  (your)  business/domain  language    

• Easy  to  both  read  and  write

Feature: Simple search As a user I want to be able to do simple searches

Scenario: I can search without any parameters Given I am on search When I perform the search Then I see the result

eBay Inc. Confidential6

Cucumber  -­‐  more  than  a  test  tool

• Behaviour-­‐Driven  Development  

• Adds  “power”  to  Gherkin  syntax  

• A  great  way  to  communicate  and  cooperate  between  “the  business  people”,  developers  and  QA.  

• Written  in  (your)  business/domain  language    

• Easy  to  both  read  and  write

Feature: Simple search As a user I want to be able to do simple searches

Scenario: I can search without any parameters Given I am on search When I perform the search Then I see the result

Step-definition!!Then(/^I see the result$/) do! unless query("view marked:'SearchResultCell'").count > 0! screenshot_and_raise "No result was shown"! end!end!!

eBay Inc. Confidential7

Calabash

• Enables  the  use  of  Cucumber  for  Native  apps  

• iOS  

• Android  

!• The  bits  and  pieces  that  interact  with  the  app  

• touch,  scroll,  swipe  etc  

• Query-­‐syntax  (iOS  examples)  

• query(“label  marked:’label  text’”)  

• query(“button  marked:’Ok’”)

eBay Inc. Confidential8

Calabash  -­‐  architecture

eBay Inc. Confidential9

Scenario  implementation  example

eBay Inc. Confidential

Scenario: Check that change classification clears matrixdata with warning! Given I am logged in as "Buyer"! And I am on the SYI hub! And I select classification "Hovedtelefoner"! And I set "Type" to “Nakkekontur”! And I set price to "250"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Fortryd"! Then I see cells containing "Nakkekontur"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Skift"! Then I see the SYI classification search!

9

Scenario  implementation  example

SYI = Sell your item

eBay Inc. Confidential

Scenario: Check that change classification clears matrixdata with warning! Given I am logged in as "Buyer"! And I am on the SYI hub! And I select classification "Hovedtelefoner"! And I set "Type" to “Nakkekontur”! And I set price to "250"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Fortryd"! Then I see cells containing "Nakkekontur"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Skift"! Then I see the SYI classification search!

9

Scenario: Check that change classification clears matrixdata with warning Given I am logged in as "Buyer" And I am on the SYI hub And I select classification "Hovedtelefoner" And I set matrixdata "Type" to “Nakkekontur" And I set price to "250"!! When I try to change classification Then I see an alert with title "Du er ved at skifte kategori" ! When I cancel the alert Then the classification is "Nakkekontur" ! When I try to change classification Then I see an alert with title "Du er ved at skifte kategori" ! When I accept the change classification alert Then I see the SYI classification search

Scenario  implementation  example

eBay Inc. Confidential10

Scenario  implementation  example

Step from “sell your item feature”!!And I set price to "250"

eBay Inc. Confidential10

Scenario  implementation  example

!And /^I set price to "(.*?)"$/ do |price|! macro 'I swipe up'! sleep(1)! touch("view marked:'Price'")! sleep(0.5)! set_text("view marked:'Price'", price)! sleep(0.5)! touch("view marked:'OK'")! sleep(1)!end

Step from “sell your item feature”!!And I set price to "250"

eBay Inc. Confidential10

Scenario  implementation  example

!And /^I set price to "(.*?)"$/ do |price|! macro 'I swipe up'! sleep(1)! touch("view marked:'Price'")! sleep(0.5)! set_text("view marked:'Price'", price)! sleep(0.5)! touch("view marked:'OK'")! sleep(1)!end

!And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end

Step from “sell your item feature”!!And I set price to "250"

eBay Inc. Confidential10

Scenario  implementation  example

!And /^I set price to "(.*?)"$/ do |price|! macro 'I swipe up'! sleep(1)! touch("view marked:'Price'")! sleep(0.5)! set_text("view marked:'Price'", price)! sleep(0.5)! touch("view marked:'OK'")! sleep(1)!end

!And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end

Step from “sell your item feature”!!And I set price to "250"

SellYourItemPage (page object) def write_price(price) scroll_and_wait_for_row_with_mark("priceCell") touch("view marked:'Price'") keyboard_enter_text price close_keyboard end

eBay Inc. Confidential11

PageObject  pattern

Feature

Scenario

Step

eBay Inc. Confidential11

PageObject  pattern

Feature

Scenario

Step

Page object

iPad iPhone Android

eBay Inc. Confidential11

PageObject  pattern

Feature

Scenario

Step

Page object

iPad iPhone Android

eBay Inc. Confidential11

PageObject  pattern

Feature

Scenario

Step

Page object

iPad iPhone Android

SyiHubPage (PageObject) !• go_to_picture_center !• go_to_classification_search !• go_to_value_selection_for_matrix

data(label) !• write_price(price)

eBay Inc. Confidential11

PageObject  pattern

Feature

Scenario

Step

Page object

iPad iPhone Android

SyiHubPage (PageObject) !• go_to_picture_center !• go_to_classification_search !• go_to_value_selection_for_matrix

data(label) !• write_price(price)

eBay Inc. Confidential12

DEMO

eBay Inc. Confidential13

Continuous  Integration

• Jenkins  on  a  couple  of  Mac  Mini’s  

• Runs  multiple  builds/tests  in  parallel  (Master/Slave  Jenkins  setup)  

• The  tests  only  runs  in  the  simulator  

• Unit  tests  —>  Integration  tests  —>  Accept/UI  tests  

• A  subset  of  the  Accept  tests  run  after  each  commit  

• Controlled  by  @tags  in  cucumber  

• All  Accept  tests  runs  once  every  evening  (or  manually)

eBay Inc. Confidential14

The  “control  room”

eBay Inc. Confidential14

The  “control  room”

eBay Inc. Confidential15

Cucumber  reports

eBay Inc. Confidential15

Cucumber  reports

eBay Inc. Confidential16

When  it  fails…

eBay Inc. Confidential17

Ensure  high  quality  tests

• Review  and/or  Pair  program  tests  with  QA  

• (Only)  use  “Custom  steps”  

• Test  code  is  (also)  production  code  

• Let  the  quality  reflect  that!  

• Refactor,  Refactor,  Refactor!  

• Make  sure  the  tests  actually  covers  what  is  expected    

• Do  it  with  facts  -­‐  no  guessing/assumptions

eBay Inc. Confidential18

Code  coverage

• Do  not  (necessarily)  focus  on  the  coverage  percentage  

• Find  inspiration  to  new  test  scenarios  

• Ensure  every  feature  in  the  app  is  well  covered

eBay Inc. Confidential18

Code  coverage

• Do  not  (necessarily)  focus  on  the  coverage  percentage  

• Find  inspiration  to  new  test  scenarios  

• Ensure  every  feature  in  the  app  is  well  covered

eBay Inc. Confidential18

Code  coverage

• Do  not  (necessarily)  focus  on  the  coverage  percentage  

• Find  inspiration  to  new  test  scenarios  

• Ensure  every  feature  in  the  app  is  well  covered

eBay Inc. Confidential18

Code  coverage

• Do  not  (necessarily)  focus  on  the  coverage  percentage  

• Find  inspiration  to  new  test  scenarios  

• Ensure  every  feature  in  the  app  is  well  covered

eBay Inc. Confidential19

Keys  to  success

• Team  effort  

• Definition  of  Done  (DoD)  includes  automated  tests  

• Know  that  it  costs  time  -­‐  it’s  not  a  silver  bullet  

• Maintenance  of  setup  -­‐  new  versions  of  iOS  &  Calabash  

• Fix  broken  builds  ASAP!

eBay Inc. Confidential20

Future  improvements  (as  seen  in  January)  

Run  a  subset  of  tests  in  Xamarin  test  cloud  a  couple  of  times  each  week    

Do  more  BDD/TDD    

Automatic  feature  builds  on  Jenkins  

Faster  feedback  cycle  

Select  a  subset  (with  @tags)  to  run  after  each  commit  

Use  “backdoor”  to  do  e.g.  login  

Avoid  resetting  after  each  scenario  

Make  all  tests  run  stable  all  the  time!

eBay Inc. Confidential21

Xamarin  Cloud  

eBay Inc. Confidential21

Xamarin  Cloud  

• Currently  only  as  proof  of  concept  on  DBA  

•  5  scenarios  on  3  devices  

• Waiting  for  a  proxy  solution  to  take  full  advantage  

• Create  listings  (and  complete  payments)  

• BilBasen  is  also  running  in  the  cloud  

• Trying  out  “Series”  for  iPad  and  iPhone  and  other  cloud  experiments  to  find  out  what  works  best  for  us

eBay Inc. Confidential22

Calabash  backdoor

eBay Inc. Confidential22

Calabash  backdoor

• Invoke  logic  in  the  app  “behind  the  scenes”  

• Reduce  scenario  test-­‐time  

• Avoid  testing  the  same  UI  over  and  over  in  each  scenario  

• Stable  

• Easy  to  use

eBay Inc. Confidential22

Calabash  backdoor

• Invoke  logic  in  the  app  “behind  the  scenes”  

• Reduce  scenario  test-­‐time  

• Avoid  testing  the  same  UI  over  and  over  in  each  scenario  

• Stable  

• Easy  to  use

• We  use  it  for  

• Login  (and  create  user)  

• Create  listings  

• Clean  up  data  

• (Code  coverage)

eBay Inc. Confidential23

Unstable  tests

eBay Inc. Confidential23

Unstable  tests

• UI  Automation  /  iOS  Simulator  often  fails  

• I  thought  it  was  Calabash  being  unstable  :-­‐)  

• We  would  start  tests  over  and  over  and  over…  

• …and  overlook  real  failures  

• It  was  almost  impossible  to  get  our  “All  scenarios”  to  run  green

eBay Inc. Confidential24

Stable  tests

• Cucumber  Rerun  formatter  to  the  rescue

eBay Inc. Confidential24

Stable  tests

• Cucumber  Rerun  formatter  to  the  rescue

A simple script that will….

eBay Inc. Confidential24

Stable  tests

• Cucumber  Rerun  formatter  to  the  rescue

Run tests - use rerun formatter

A simple script that will….

eBay Inc. Confidential24

Stable  tests

• Cucumber  Rerun  formatter  to  the  rescue

Run tests - use rerun formatter

If rerun.txt contains tests - run them

A simple script that will….

eBay Inc. Confidential24

Stable  tests

• Cucumber  Rerun  formatter  to  the  rescue

Run tests - use rerun formatter

If rerun.txt contains tests - run them

Inform jenkins how it all went

A simple script that will….

eBay Inc. Confidential24

Stable  tests

• Cucumber  Rerun  formatter  to  the  rescue

Run tests - use rerun formatter

If rerun.txt contains tests - run them

Inform jenkins how it all went

A simple script that will….

Replace rerun result in original result

eBay Inc. Confidential24

Stable  tests

• Cucumber  Rerun  formatter  to  the  rescue

Run tests - use rerun formatter

If rerun.txt contains tests - run them

Inform jenkins how it all went

A simple script that will….

Replace rerun result in original result

eBay Inc. Confidential25

Wrap  up!

• Test  scenarios  follows  the  code  =  the  truth  

• Enables  rapid  releases  /  fast  time  to  market  

• A  necessity  to  deliver  high  quality  

• Spend  less  time  doing  (boring)  trivial  regression  tests  

• Happy  QA´er  

• Brave  developers  -­‐  higher  code  quality  

• Quality  product  

• Happy  users

eBay Inc. Confidential26

QUESTIONS?

eBay Inc. Confidential27

Ressourcer  

• The  Cucumber  Book    

• Calabash  @github  (https://github.com/calabash/)  

• Calabash  google  groups    • https://groups.google.com/forum/#!forum/calabash-­‐ios  

• https://groups.google.com/forum/#!forum/calabash-­‐android    

• Calabash  overview  (http://docs.xamarin.com/guides/cross-­‐platform/test_cloud/calabash/intro_to_calabash/)  

• Jenkins  (http://jenkins-­‐ci.org/)  • plugins:  Cucumber-­‐reports,  Cobertura  (code  coverage),  Build  monitor  view  

• RubyMine  (http://www.jetbrains.com/ruby/)  

• Uncle  Bobs  Clean  Code    

• because  it’s  a  great  book  that  promotes  craftsmanship