Magento Testing on all fronts

On all Fronts Testing

Magento

San Francisco Magento Meetup Group - March 11, 2014

Fabrizio Branca Lead System Developer at

E-Commerce: Magento

CMS: TYPO3

Portals: ZF, FLOW,…

Mobile Searchperience: SOLR

120 people in 7 offices world-wide

High Performance

/Scale

Global Enterprise Projects

Unit

Service

UI Acceptance

http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid

What the customer thinks

the code should do

What the developer thinks

the code should do

Load Test

Continuous Integration

Deployment

Development

Delivery

Improvement

Quality Assurance Continuous

Everything sounds better with “continuous”

“Continuous” == “Doing stuff over and over again”

Builds Deployment Tests Infrastructure

Setup

Devbox Setup

== “Automation”

tim

e s

pe

nt

task size

does it manually

does it manually

gets annoyed

writes script to automate

runs script

wins

loses

Geeks vs. Non-Geeks

Makes fun of geek’s complicated method

— Ovidiu Platon

We are not shipping your machine! if it works on your machine! I don’t care

Picture The BIG

Vagrant Boxes

Awesome stuff

Every single line potentially

can break production!

Bratwurst (Germans)

Donut (Americans)

Pizza (Italians)

Version Control

GIT, SVN,…

PHP lint

Version Control Code Reviews

Gerrit, Crucible, Phabricator,…

Version Control Code Reviews Build

Files

DB

Installer

Settings

Triggered by commit, by time

or manually

Jenkins

minify js/css

.tar.gz

“virtual package” (revision)

Version Control Code Reviews Build

Files

DB

Installer

Settings

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

CodeSniffer

php-pmd

php-cpd

php-depend

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration”

First system where code from all

developers/teams meets each other

unstable system. Might be broken. Will be rebuild

without warning

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests

PHPUnit

Code coverage

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests

Test features that rely on other parts (e.g. framework, external

services,…) External components (SSL, Redis, Varnish,…)

Basically this is also PHPUnit, but

addressing other stuff

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Behat (Cucumber)

Same setup as production, but different machines

Client QA / approval

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning

Stress

Tests

Jmeter

Spin up EC2 instances in the cloud to produce

traffic*

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Stress

Tests

Install on

“Production”

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Stress

Tests

Install on

“Production” Enable Features

Version

Control Code

Reviews Build Static Code

Analysis

Install on

“Integration”

Unit Tests Integration

Tests

Acceptance

Tests

Install on

“Staging”

Install on

“Production” Enable

Features

with EcomDev_PHPUnit Unit Tests

https://github.com/EcomDev/EcomDev_PHPUnit

use the “dev”

branch!

Config Model Block Helper Controller

fat model, skinny controller…

Classes PHPUnit_Framework_TestCase EcomDev_PHPUnit_Test_Case

EcomDev_PHPUnit_Test_Case_Config EcomDev_PHPUnit_Test_Case_Controller

Assertions

Fixtures

Mocks

Data Provider

Magento Integration

Testsuite management

Framework integration

Create test environment (db+config)

Manual

http://www.ecomdev.org/2011/05/24/ecomdev_phpunit-manual-version-0-2-0-is-available-for-

download.html

Video tutorial

http://www.ecomdev.org/2012/01/31/test-driven-magento-development-seminar-video-and-

code.html

Documentation More Information

With Selenium Acceptance Testing

Acceptance Tests

What’s out there? Selenium for PHP

PHPUnit_Selenium (Sebastian Bergmann)

WebDriver-PHP (Chibimagic)

Php-webdriver-bindings (Lukasz Kolczynski)

php-webdriver

PHPUnit_Selenium (Giorgio Sironi)

php-webdriver Selenium 2

08/2011 Facebook/php-webdriver (Justin Bishop)

Element-34/php-webdriver (Adam Goucher)

instaclick/php-webdriver (Anthon Pang)

php-webdriver

“A very thin wrapper of WebDriver”

So what?

A thin layer good! But not enough…

no test logic

no higher level components

dealing with Selenium specific tasks

not compatible to legacy test cases

No abstraction / structure

Architecture

HTTP AUT

No direct access (DB, classes,…)

REST* (over HTTP)

*http://code.google.com/p/selenium/wiki/JsonWireProtocol

Project test cases

PHPUnit

Project Components

Components Library

Menta

php-webdriver

reusable

project specific

Menta Selenium 2 Testing Framework for PHP

Who is “Menta”?

What is “Menta”?

Features Menta The framework.

Not the cat…

Components Session Management

? Bonus Features :) Event / Observer Configuration Management

PHPUnit integration

Why?

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

Data Providers

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

Dependent Tests

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

Test Infrastructure

Menta for PHPUnit Abstract test class

PHPUnit_Framework_TestCase Menta_PHPUnit_Testcase_Selenium2 ProjectY_Abstract_TestCase

Very thin layer on top of PHPUnit

ProjectY_TestCase_Registration ProjectY_TestCase_Checkout the “magic” happens in the components

HTML Report Screenshots (on demand

and on failure)

test description from phpdoc

additional messages

error details

HTML Report “Duration Heatmap”

shows long running tests

Menta for PHPUnit Text Result Printer

PHPUnit 3.7.20 by Sebastian Bergmann. Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml > SUITE: > SUITE: > SUITE: Acceptance_Tests_Account_Register > TEST: Acceptance_Tests_Account_Register::register SUCCESS. (Duration: 8.18 sec) > TEST: Acceptance_Tests_Account_Register::login SUCCESS. (Duration: 6.5 sec) > TEST: Acceptance_Tests_Account_Register::testRegistrationMail FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22 sec) < Duration: 36 sec < Duration: 36 sec > SUITE: > SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest > TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Failed asserting < Duration: 9.87 sec > SUITE: Acceptance_Tests_Checkout_CheckoutTest > TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView SUCCESS. (Duration: 1.79 sec) > TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::checkout" to pass. (Duration: 5.34 sec)

Components

Common Assert Wait Screenshot Selenium1Facade

Imap WebService W3C Validator Curl …

Some might not even require a Selenium session

Components! PageObjects?

PageObjects Components

{ } }⊇{

PageObjects are

components

PageObjects?

represents services offered by a page or feature

is the only thing with a deep knowledge of

the HTML

clickCheckoutButton() isInCart($product) getTotals()

✔ ✔ ✔

Component Libraries

Menta

Magento

(your library)

Project-

specific

Magento

(community-provided)

General

(your library)

Gallery Screenshot

Perceptual diffs Build X

Build X-1

Perceptual Diff

github.com / AOEmedia / Menta github.com / AOEmedia / Menta_SampleProject

It’s composer based and you’ can get started in a few minutes!

With JMeter Load Testing

JMeter

jmeter.apache.org Current version:

2.11

JMeter HTTP

Simple Setup

JMeter

Slave

JMeter

Slave

JMeter

Slave

JMeter

Master

Master/Slave Setup

JMeter

Slave

JMeter

Slave

JMeter

Slave

JMeter

Master

EC2 Setup

JMeter

Slave

JMeter

Slave

…

Creating JMeter tests

JMeter

Test

Manually

Replaying Apache Logs

Recording Browser

interaction using a proxy “Convert” existing

acceptance tests

CSV Files

(Selenium)

HTTP

HTTP Behat Selenium Firefox Proxy

JMeter

Test

Converting Behat to JMeter

Using a Proxy JMeter

Using a Proxy Browser configuration

Manual configuration

Browser Extension

Using a Proxy Browser configuration

java -jar selenium-server-standalone-2.40.0.jar

-trustAllSSLCertificates

-Dwebdriver.firefox.profile=jmeter

Using a Proxy Selenium configuration

http://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf

https://wiki.apache.org/jmeter/TestRecording210

http://jmeter.apache.org/usermanual/component_reference.html#HTTP%28S%29_Test_Script

_Recorder

Using a Proxy

https://jmeter.apache.org/usermanual/test_plan.html

• ThreadGroups

• Controllers

• Samplers

• Logic Controllers

• Test Fragments

• Timers

• Assertions

• Configuration Elements

• Pre-Processor Elements

• Post-Processor Elements

Elements of a Test Plan

http://www.slideshare.net/shubelal/abj-meterbrief-upladed/7

Elements of a Test Plan

• Tree structure

• Inheritance (Cookie Manager, Header Manager,…)

• Timers

• Variables

• Assertions

• Random values

• Scripting

• Extracting values

• Following redirects

• (check examples in bin/examples)

Basic Concepts

Workflows Search / Filters Static Pages

404

What to test Consider conversion rate /

funnel!

JMeter Plugins

jmeter-plugins.org

Distributed load testing

General Master/Slave setup

https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf

AOE presentations

http://www.slideshare.net/axeljung01/jmeter-and-amazon-cloud-workshop-t3dd2012

http://www.slideshare.net/aoemedia/performance-measurement-and-tuning

On EC2

https://github.com/oliverlloyd/jmeter-ec2

http://www.http503.com/2012/run-jmeter-on-amazon-ec2-cloud/

Server Monitoring

Thank you! Any questions?

http://www.aoe.com

http://www.fabrizio-branca.de

@fbrnc Follow me on twitter!

My blog