Upload
harlan
View
42
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Test - Strategy and Implementation. By jonasbn . What this presentation will cover?. An outline of test methods How to design a test strategy from test methods How to implement it a test strategy using test methods Exemplified by all the mistakes humanly possible. - PowerPoint PPT Presentation
Citation preview
Test - Strategy and Implementation
Test - Strategy and Implementation
By jonasbn <[email protected]>
Test - Strategy and Implementation
What this presentation will cover?
• An outline of test methods
• How to design a test strategy from test methods
• How to implement it a test strategy using test methods
• Exemplified by all the mistakes humanly possible
Test - Strategy and Implementation
What this presentation will not give you?
• A solution to all your problems
• A neat Solution::Simple for download from CPAN
Test - Strategy and Implementation
What this presentation try to give you?
• A new mindset when thinking about testing
• A tool to make better tests
• A tool to make the right tests
• A tool to improve your development
Test - Strategy and Implementation
Test
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
Test - Strategy and Implementation
Are YOU a Tester?
“Hello my name is jonasbn – I’m a tester. In my daily work I work as a developer and that’s what it says on the paper, but I’m really a tester. Every time my colleagues turn the other way I write one and every time my manager gives me a task - all I think about is tests and how to test – often I work late so I can finish my tests without using valuable development time – Yes I’m a tester and I cannot help it.” - jonasbn, developer (tester)
Test - Strategy and Implementation
What does test mean?
• How do you define test?
Test - Strategy and Implementation
My test definition
• Tests can assure me that what I am coding works correctly
• Tests are aiming at being automated
• Test is a tool to help my development
• Tests can be difficult to develop correctly
Test - Strategy and Implementation
What kind of test methods do you know?
• What test methods do you currently use?
• Or have discarded?
• What test methods have you used?
Test - Strategy and Implementation
My list of Test Methods
• Coverage Test• Smoke Test• Integration Test• Acceptance Test• System Execution Test• Regression Test• Stress Test• Performance Test• Unit Test
Test - Strategy and Implementation
Coverage Test
• Simple Philosophy– All lines should be run at least once– Testing all possible branches of the code– Time consuming– Large number of tests– Difficult way of testing– Tools to assist coverage test exist
Test - Strategy and Implementation
Smoke Test
• From the world of electronics– Power it up - does it break?– Not much detail– Crude way of testing
Test - Strategy and Implementation
Integration Test
• When things need to be assembled– Known from large projects– For dispersed development– When integration with 3rd. Party components
Test - Strategy and Implementation
Acceptance Test
• When the Client/Customer are to approve– Should not replace Performance Test– Should not replace automated testing– Should not be automated - this is a manual
process
Test - Strategy and Implementation
System Execution Test
• From phase based development– When test is a phase– Focus is changed– Many aspects covered in other methods– Has serious flaws– Can be cut due to time constraints
Test - Strategy and Implementation
Regression Test
• When old things are revisited– Used to ensure backwards compability
Test - Strategy and Implementation
Stress Test
• When trying to break the system– Resembles smoke test– Also known as monkey test– Changing environments– Garbage Feeding
Test - Strategy and Implementation
Performance Test
• When speed is the need…– For benchmarking– Can be used to identify bottlenecks– History is important
Test - Strategy and Implementation
Unit Test
• The foundation of all testing– Can be used to implement almost all test
methods– Simplicity– SMOP– Can be automated– Programmatic approach
Test - Strategy and Implementation
Test Methods Summarized
• Automatic and manual test methods
• Black box and White box angles
Test - Strategy and Implementation
Black/White Box Testing
• Nothing is known apart from what is public
• Everything is known• Introspections are
possible
Test - Strategy and Implementation
Test StrategyMelchett: You look surprised, Blackadder. Edmund: I certainly am, sir. I didn't realise we had any battle plans.Melchett: Well, of course we have! How else do you think the battles are directed?Edmund: Our battles are directed, sir?Melchett: Well, of course they are, Blackadder -- directed according to the Grand Plan.Edmund: Would that be the plan to continue with total slaughter until everyone's dead except Field Marshal Haig, Lady Haig and their tortoise, Alan?Melchett: Great Scott! Even you know it! Guard! Guard! Bolt all the doors; hammer large pieces of crooked wood against all the windows! This security leak is far worse than we'd imagined!
Test - Strategy and Implementation
Project Goals
• We need to examine the project’s goals to make the right test strategy
• The test strategy should support the project’s goals not work against them
Test - Strategy and Implementation
Defining a strategy
• We apply a selection the listed test methods to form a strategy– Which ones should we pick?
Test - Strategy and Implementation
An example - “eXtreme Programming”
• Unit Test during development
• Integration Test when applicable
• Acceptance Test and the end of each cycle
Test - Strategy and Implementation
An example - “Waterfall/Phase-divided”
• Unit Test during development
• Smoke Test when building
• Integration Test when partially done
• System Execution Test when done
• Acceptance Test
Test - Strategy and Implementation
Other Strategies?
• What strategy do YOU use?
Test - Strategy and Implementation
Select the right strategy
• Is your application a GUI?
• Is your application performance critical?
• Is quality important?
• Is data correctness essential?
• Is data processing essential?
• Is backwards compability important?
Test - Strategy and Implementation
Test Implementation
Test - Strategy and Implementation
Basic Unit Tests
use strict;use Test::More tests => 6;use lib qw(lib);
BEGIN { use_ok( 'Geneva::Product' ); }require_ok( 'Geneva::Product' );
my $p = Geneva::Product->new();ok(! (ref $p), 'failing constructor test');
$p = Geneva::Product->new(p_customerRef => 'argle',
);ok((ref $p) eq 'Geneva::Product', 'succeeding constructor test');ok($p->isa('Geneva'), 'Testing isa of Geneva');ok($p->can('new'), 'Testing can Geneva::Product::new');
Test - Strategy and Implementation
Unit Test Tactics
• Should I go for black box or white box?
• Can I trust the units
Test - Strategy and Implementation
Advanced Unit Tests 1
sub move2done ($) {
my $file = shift ;
if ( -e "$mail_processed/$file") {
unlink ("$mail_processed/$file");
}
my $rv = rename("$mail_spool/$file","$mail_processed/$file") || warn "Unable to move file: $mail_spool/$file to$mail_processed/$file - $!\n";
return $rv;
}
Test - Strategy and Implementation
Advanced Unit Test 2sub DSL0281ReceiveTDCRejected ($$) {
my ($dbh, $filename) = @_;my ($dslid, $order_copper_id, $reason) = RejectedHandler($filename);
if ($dslid and $order_copper_id) {
eval {my $status = getstatus($dbh, $order_copper_id);if ($status ne DB_STATUS_RECEIVED) {die ("Status of order_copper_id $order_copper_id is $status, expected ".DB_STATUS_RECEIVED) ;}updateOrderCopper($dbh, $order_copper_id, DB_STATUS_REJECTED, undef, undef, \$reason);addtolog($dbh, DB_STATUS_REJECTED, $dslid, $order_copper_id);} ;
if ($@) { warn $@ ; return "Error: $@" ;} else { return "OK" ; }
} else { return "Error: DSL_ID or Order_Copper_Id could not be extracted from mail" ;
}}
Test - Strategy and Implementation
Acceptance Test
• Acceptance Test is a manual process and should therefore not be automated
• Unit Test reports can add some color to the delivery though underlining the fact that the product/system has been tested
Test - Strategy and Implementation
Coverage Test
• Devel::Coverage
• Devel::Cover
Test - Strategy and Implementation
Smoke Test
• Perl smoke testing worth a study
Test - Strategy and Implementation
Integration Test
• You can easily test this if you take a Smoke Test approach
Test - Strategy and Implementation
System Execution Test
• This Test method is merely a phase, but it should be implemented using the test methods/strategy mentioned in this presentation
Test - Strategy and Implementation
Regression Test
• Regression Test will be necessary at some point
• Save and use old tests, don’t throw anything away
Test - Strategy and Implementation
Stress Test
• Just bombard your application with calls from your test suite
• Your test suite will cover “all” aspects of your application
• Do it structured
• Examine the output
Test - Strategy and Implementation
Performance Test 1
• Benchmark is your friend
• Save reports, historic data is essential
• Examine output
• And compare output
Test - Strategy and Implementation
Performance Test 2use SOAP::Lite;use Benchmark;
my $t0 = new Benchmark;my $proxy = "http://somemachine/tempuri/Service1.asmx";my $uri = "http://service.domain.dk/";
my $soap = SOAP::Lite->uri($uri)->proxy($proxy)->on_action(sub {join '', @_});
my $method = SOAP::Data->name('getcustomer')->attr({xmlns => 'http://service.domain.dk/'});
my @params = ();push(@params, SOAP::Data->name('dslid' => 'dsl89305')->attr({'xsi:type' =>
'xsd:string'}));
my $rc = $soap->call($method => @params)->valueof("//getcustomerResponse");my $t1 = new Benchmark;my $td = timediff($t1, $t0);print "the code took:",timestr($td),"\n";
Test - Strategy and Implementation
Databases
• Database encapsulations can be tricky to test
• Data is often volatile and a baseline is difficult to optain
Test - Strategy and Implementation
Solution 1: “The Sims”
• The volatile aspect disappears
• Data is known and controlled
• The data can be misleading
Test - Strategy and Implementation
Solution 2: “The Others”
• Real-life data
Test - Strategy and Implementation
“The Others Example”use strict;use Test::More tests => 2;use LWP::UserAgent;
my $phoneno = '32545697';my $uri = 'http://hostname.org/cgi-bin/proxy1.cgi';$uri .= "?phoneno=$phoneno";
my $ua = LWP::UserAgent->new(env_proxy => 1, keep_alive => 1, timeout => 30,);
my $request = HTTP::Request->new('GET', $uri);my $response = $ua->request($request);my $content = $response->content;
is($response->message, "OK", "Testing connection to URL: $uri");like($content, qr/Amagerbro/i, "Testing content of response");
Test - Strategy and Implementation
The Frame-work
• Too many tests
• Redundant code all over the place
• Programmatic Practice
• SMOP
Test - Strategy and Implementation
The Frame-work 2my %data_map = (
message => {type => 'xsd:string',value => ’kødpålæg',
},);my $test_target = 'addtolog';
my $proxy = "http://webservice/Service1.asmx";my $uri = "http://webservice.host.org/";my @api_params = qw(message);
my $t = Test::Accept::SOAP->new();$t->test(
'proxy' => $proxy,'uri' => $uri,'test_target' => $test_target,'api_params' => \@api_params,'data_map' => \%data_map,
);
Test - Strategy and Implementation
The Frame-work 3sub test ($%) {
my ($self, %args) = @_;
my $proxy = $args{'proxy'};my $uri = $args{'uri'};my $test_target = $args{'test_target'};my $api_params = $args{'api_params'};my $data_map = $args{'data_map'};my $soap = $args{'soap'};
unless ($proxy and $uri and $test_target and $api_params and $data_map) {warn "Unable to perform test, insufficient parameters\n"; return;
}my $method = $self->methodize($uri, $test_target);my @params = ();foreach my $p (@{$api_params}) { $self->parameterize(\@params,$data_map, $p); }my $rc = $self->exercize($soap, $test_target, $method, \@params, $debug);ok($rc, 'Checking we got something back from the webservice');my $test_result = $test_target.'Result';ok($rc->{$test_result}, 'Testing the returned result (shallow)');
return 1;}
Test - Strategy and Implementation
Test Related Perl Modules
• Test
• Test::Simple
• Test:More
• Devel::Cover
• Devel::Coverage
Test - Strategy and Implementation
The Power of Perl
• SOAP/Web Services
• SNMP
• SMTP
• HTTP
• Databases
• Whatever you can think of…
Test - Strategy and Implementation
References
• ‘The C++ Programming Language’ 2nd. Ed., Bjarne Stroustrup
• ‘Extreme Programming Explained – embrace change’, Kent Beck
• ‘Code Complete’, Steve McConnell• ‘Testing and Code Coverage’ abstract, Paul Johnson• ‘Test::More’ POD, Michael Schwern• ‘Test - Strategy and Implementation’ article,
jonasbn