SmokeTestsWhy you should try to burn down
your production environment
Sebastian ThoßChapter Lead Backend
Disclaimer
Agenda• Project Architecture
• Types Of Tests
• SmokeTests
• Server Architecture
• Questions & Answers
Project Architecture
"one hundred fifty-seven quinvigintillion, seven hundred eighty-six quattuorvigintillion, six hundred fifty-seven trevigintillion, seven hundred
forty-seven duovigintillion, three hundred forty unvigintillion, one hundred eighty-six vigintillion, (…) nine hundred forty-five quintillion, eight hundred
twenty-eight quadrillion, two hundred seventy trillion, eighty billion, …"
Webserver
Key-Value Store
Search
FURY Frontend FURY Backend
FURY Components
Webserver
Key-Value Store
Search
FURY Frontend FURY Backend
Legacy RDBMSLegacy System Session Storage
Collect & Export
K-VStore
Webserver
Key-Value Store
Search
FURY Frontend FURY Backend
Legacy RDBMSLegacy System
FURY Requests
200 OK
Session Storage
K-VStore
Webserver
Key-Value Store
Search
FURY Frontend FURY Backend
Legacy RDBMSLegacy System
Requests to Legacy System
404 NOT FOUND
Session Storage
K-VStore
Webserver
Key-Value Store
Search
FURY Frontend FURY Backend
Legacy RDBMSLegacy System
Requests to Legacy System
404 NOT FOUND
200 OK
Session Storage
K-VStore
More information on www.sebastianthoss.de
Types Of Tests
Test Pyramid
UNIT TESTS
Class 2
Class 3
Mock
Mock
Unit Tests
Class 1
Class 4
Test Pyramid
INTEGRATION TESTS
UNIT TESTS
Class 4Class 3
Integration Tests
Class 1 Class 2
Mock Mock
Test Pyramid
INTEGRATION TESTS
UNIT TESTS
100% CODE COVER
AGE
Test Pyramid
ACCEPTANCE TESTS
INTEGRATION TESTS
UNIT TESTS
Acceptance Tests
Class 1 Class 2
Class 3 Class 4
Test Pyramid
ACCEPTANCE TESTS
INTEGRATION TESTS
UNIT TESTS
?
SmokeTests
What are SmokeTests?
In computer programming and software testing, smoke testing is preliminary testing to reveal simple failures severe enough to reject a prospective software release. Source: https://en.wikipedia.org/wiki/Smoke_testing
What are SmokeTests?
SmokeTests should…• … be simple
• … be fast
• … test pages with optional parameters too
• … cover at least all URLs in google index
• … use a manual maintained list of URLs
How do SmokeTests work?
https://www.my-application.com/foo
<html><body>…</body></html>
TTFB: 65ms
HTTP 1.1/200 OK
SmokeTestClient
CI Server
Application to test
Production Server
How do SmokeTests work?
https://www.my-application.com/foo
<html><body>…</body></html>
TTFB: 320ms
HTTP 1.1/200 OK
SmokeTestClient
CI Server Production Server
Application to test
What should SmokeTests validate?
• Status code
• Time to first byte
• If body is provided
• Correct server
SmokeTests are NOT Acceptance Tests
SmokeTestClient
HTTP 1.1/200 OK
<html> <head> <title>Foo</title> <body> <div id="bar"><span>foobar</span></div> </body> </html>
namespace Kartenmacherei\Testing;
use PHPUnit_Framework_TestCase;
class SmokeTest extends PHPUnit_Framework_TestCase { /** * @dataProvider furyUrlProvider * * @param Url $url */ public function testFuryUrl(Url $url) { $result = $this->sendGetRequest($url);
$this->assertSame(200, $result->getStatusCode()); $this->assertNotEmpty($result->getBody()); $this->assertLessThanOrEqual(100, $result->getTimeToFirstByteInMilliseconds()); }
public function furyUrlProvider() { $urls = ['http://www.kartenmacherei.de', …]; $urlCollection = UrlCollection::fromStrings($urls);
return $urlCollection->asDataProviderArray($urlCollection); }
How to speed up SmokeTests?
Concurrent SmokeTests
SmokeTestClient
CI Server
Applicationto test
Production Server
https://www.my-application.com/baz
https://www.my-application.com/bar
https://www.my-application.com/foo
Concurrent SmokeTests
https://www.my-application.com/baz
https://www.my-application.com/bar
https://www.my-application.com/foo
TTFB: 34ms
SmokeTestClient
CI Server
Applicationto test
Production Server
Concurrent SmokeTests
https://www.my-application.com/foobaz
https://www.my-application.com/bar
https://www.my-application.com/foo
SmokeTestClient
CI Server
Applicationto test
Production Server
Concurrent SmokeTests
TTFB: 65ms
https://www.my-application.com/foobaz
https://www.my-application.com/bar
https://www.my-application.com/foo
SmokeTestClient
CI Server
Applicationto test
Production Server
Concurrent SmokeTestshttps://www.my-application.com/123
https://www.my-application.com/foobaz
https://www.my-application.com/bar
SmokeTestClient
CI Server
Applicationto test
Production Server
Concurrent SmokeTests
TTFB: 620ms
https://www.my-application.com/123
https://www.my-application.com/foobaz
https://www.my-application.com/bar
SmokeTestClient
CI Server
Applicationto test
Production Server
Source: http://www.ve7kfm.com/fcc-server.jpg
Let’s try to burn it down!
There is a package for SmokeTests
DjThossi/Smoke-Testing-PHP
DataProvider
Test
Application
HTTP Requests
HTTP Responses
PHPUnit
Call
Result[]
Result
class SmokeTest extends PHPUnit_Framework_TestCase { use SmokeTestTrait;
/** * @dataProvider myDataProvider */ public function testExample(Result $result) { $this->assertSuccess($result); $this->assertTimeToFirstByte(new TimeToFirstByte(100), $result); $this->assertBodyNotEmpty($result); $this->assertHeaderExists(Header::fromPrimitives(‘App-Server', ‘Fury’), $result); }
public function myDataProvider() { $urls = ['http://www.kartenmacherei.de', …]; $options = new SmokeTestOptions( UrlCollection::fromStrings($urls), new RequestTimeout(2), new FollowRedirects(true), new Concurrency(3), new BodyLength(500) );
return $this->runSmokeTests($options); }
class SmokeTest extends PHPUnit_Framework_TestCase { use SmokeTestTrait;
/** * @dataProvider myDataProvider */ public function testExample(Result $result) { $this->assertSuccess($result); $this->assertTimeToFirstByte(new TimeToFirstByte(100), $result); $this->assertBodyNotEmpty($result); $this->assertHeaderExists(Header::fromPrimitives(‘App-Server', ‘Fury’), $result); }
public function myDataProvider() { $urls = ['http://www.kartenmacherei.de', …]; $options = new SmokeTestOptions( UrlCollection::fromStrings($urls), new RequestTimeout(2), new FollowRedirects(true), new Concurrency(3), new BodyLength(500) );
return $this->runSmokeTests($options); }
class SmokeTest extends PHPUnit_Framework_TestCase { use SmokeTestTrait;
/** * @dataProvider myDataProvider */ public function testExample(Result $result) { $this->assertSuccess($result); $this->assertTimeToFirstByte(new TimeToFirstByte(100), $result); $this->assertBodyNotEmpty($result); $this->assertHeaderExists(Header::fromPrimitives(‘App-Server', ‘Fury’), $result); }
public function myDataProvider() { $urls = ['http://www.kartenmacherei.de', …]; $options = new SmokeTestOptions( UrlCollection::fromStrings($urls), new RequestTimeout(2), new FollowRedirects(true), new Concurrency(3), new BodyLength(500) );
return $this->runSmokeTests($options); }
class SmokeTest extends PHPUnit_Framework_TestCase { use SmokeTestTrait;
/** * @dataProvider myDataProvider */ public function testExample(Result $result) { $this->assertSuccess($result); $this->assertTimeToFirstByte(new TimeToFirstByte(100), $result); $this->assertBodyNotEmpty($result); $this->assertHeaderExists(Header::fromPrimitives(‘App-Server', ‘Fury’), $result); }
public function myDataProvider() { $urls = ['http://www.kartenmacherei.de', …]; $options = new SmokeTestOptions( UrlCollection::fromStrings($urls), new RequestTimeout(2), new FollowRedirects(true), new Concurrency(3), new BodyLength(500) );
return $this->runSmokeTests($options); }
class SmokeTest extends PHPUnit_Framework_TestCase { use SmokeTestTrait;
/** * @dataProvider myDataProvider */ public function testExample(Result $result) { $this->assertSuccess($result); $this->assertTimeToFirstByte(new TimeToFirstByte(100), $result); $this->assertBodyNotEmpty($result); $this->assertHeaderExists(Header::fromPrimitives(‘App-Server', ‘Fury’), $result); }
public function myDataProvider() { $urls = ['http://www.kartenmacherei.de', …]; $options = new SmokeTestOptions( UrlCollection::fromStrings($urls), new RequestTimeout(2), new FollowRedirects(true), new Concurrency(3), new BodyLength(500) );
return $this->runSmokeTests($options); }
class SmokeTest extends PHPUnit_Framework_TestCase { use SmokeTestTrait;
/** * @dataProvider myDataProvider */ public function testExample(Result $result) { $this->assertSuccess($result); $this->assertTimeToFirstByte(new TimeToFirstByte(100), $result); $this->assertBodyNotEmpty($result); $this->assertHeaderExists(Header::fromPrimitives(‘App-Server', ‘Fury’), $result); }
public function myDataProvider() { $urls = ['http://www.kartenmacherei.de', …]; $options = new SmokeTestOptions( UrlCollection::fromStrings($urls), new RequestTimeout(2), new FollowRedirects(true), new Concurrency(3), new BodyLength(500) );
return $this->runSmokeTests($options); }
Output
Features to come next
• Improve ErrorResult object
• Introduce assertions for Redirects
• Improve quality of error message
• 2nd version based on PHPUnit 6
Server Architecture
Webserver (Router)
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
Webserver
FURY Frontend
Server B
K/V StoreSearch
FURY Backend
Webserver (Router)
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
Webserver
FURY Frontend
Server B
K/V StoreSearch
FURY Backend
active = A
Webserver (Router)
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
Webserver
FURY Frontend
Server B
K/V StoreSearch
FURY Backend
active = B
Webserver (Router)
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
active = B
RDBMS Read Slave
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
Webserver (Router)
Build Server
Deploy Code
active = B
RDBMS Read Slave
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
Webserver (Router)
Build Server
Deploy Code
Collect & Export
active = B
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
Webserver (Router)
Build Server
Deploy Code
Collect & Export
Smoke Tests
active = B
RDBMS Read Slave
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
Webserver (Router)
Build Server
Deploy Code
Collect & Export
Smoke Tests
active = B
RDBMS Read Slave
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
Webserver (Router)
Build Server
Deploy Code
Collect & Export
Smoke Tests
active = B
RDBMS Read Slave
Webserver
FURY Frontend
Server A
K/V StoreSearch
FURY Backend
Webserver (Router)
Build Server
Deploy Code
Collect & Export
Smoke Tests
Switch to A
active = Bactive = A
RDBMS Read Slave
Conclusion
• Write tests
• Get 100% coverage
• SmokeTest your Website
• Only activate server if it didn’t start smoking
https://www.facebook.com/kartenmacherei/
http://inside.kartenmacherei.de/job.html
https://tech.kartenmacherei.de
@techdotkam
Contact & Feedback
Q&A
Recommended
![[500DISTRO] Measuring for Engagement: Understanding User Gains, Losses & Levels of Interaction](https://img.pdfslide.us/doc/110x75/53f8b2c38d7f729c2e8b47cf/500distro-measuring-for-engagement-understanding-user-gains-losses-levels-of-interaction.jpg)
![[#500Distro] Heating Up the Open Rate: New School Rules for Email Marketing](https://img.pdfslide.us/doc/110x75/53f8b3788d7f72b82e8b4795/500distro-heating-up-the-open-rate-new-school-rules-for-email-marketing.jpg)
![[500DISTRO] Mind Over Matter: Tactics for Testing Assumptions & Increasing Conversion](https://img.pdfslide.us/doc/110x75/53f8b3c98d7f729c2e8b47d3/500distro-mind-over-matter-tactics-for-testing-assumptions-increasing-conversion.jpg)
![[500DISTRO] Dissecting UX: Funnel Fixes & Other CRO Tips for Growth That Pays](https://img.pdfslide.us/doc/110x75/53f8b3658d7f72b82e8b4794/500distro-dissecting-ux-funnel-fixes-other-cro-tips-for-growth-that-pays.jpg)
![[#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test](https://img.pdfslide.us/doc/110x75/53f8b2358d7f72b82e8b4791/500distro-measuring-for-impact-knowing-when-what-how-to-ab-test.jpg)
