Positive and Negative Testing

8/3/2019 Positive and Negative Testing

1/7

Positive and Negative Testing ____________________________

by Jeff Nyman.GlobalTester, TechQA, Copyright 2002 All Rights Reserved.

The notion of something like "Integration Testing" or "System Testing" can (andshould) be defined so that everyone knows what is meant by that activity withinthe same organizatoin, but terms like "negative test" and "positive test" aremore of a concept than a strict activity. In both instances you are dealing withan input, an action, and an output. The action acts upon the input to derive acertain output. So a test case (and thus a good test) is just one that deals withthose three things. Both test cases can produce errors and, in fact, some saythat the success of a test case is based upon the probability of it finding newerrors in an application.

What I want to do here, however, is state clearly one viewpoint of what thedistinction between positive and negative testing is. Then I want to play Devil'sAdvocate and try to undermine that viewpoint by presenting an argument thatothers have put forth - an alternative viewpoint. The real point of this will be toshow that sometimes trying to adhere too rigidly to conceptual terms like this

can lead to a lot of stagnating action. Read this section as a sort of extendedargument that I am having with myself as I come to grips with these terms.

So let us first state a simple hypothetical definition: positive testing is thattesting which attempts to show that a given module of an application does whatit is supposed to do. Negative testing is that testing which attempts to show thatthe module does not do anything that it is not supposed to do. So, by that logic,and to make a concrete example, an application delivering an error when itshould is actually an example of a positive test. A negative test would be theprogram not delivering an error when it should or delivering an error when itshould not. But this sounds like it is more based on what the application doesduring testing rather than how the tester is actually going about testing it. Well,sort of. The idea here is that neither test necessarily has to force an errorcondition, per se, at least by strict definition. But both concepts (negative andpositive) are looking for different types of error conditions. Consider that one

part of negative testing is often considered to be boundary analysis. In thiscase, you are not so much "forcing an error" because, of course, the applicationshould handle boundary problems. But what you are doing is seeing if theboundary problem is not, in fact, handled. So if the program is supposed to givean error when the person types in "101" on a field that should be between "1"and "100", then that is valid if an error shows up. If, however, the applicationdoes not give an error when the user typed "101" then you have a problem. Soreally negative testing and positive testing are the same kinds of things whenyou really boil it right down.

Now, some make a distinguishing remark from what I said. I said the following:

Positive testing is that testing which attempts to show that a given module of anapplication does what it is supposed to do.Negative testing is that testing which attempts to show that the module does notdo anything that it is not supposed to do.Playing the Devil's Advocate, others would change this around and say thefollowing is a better distinction:

Positive testing is that testing which attempts to show that a given module of anapplication does not do what it is supposed to do.Negative testing is that testing which attempts to show that the module doessomething that it is not supposed to do.Let us look at this slightly shifted point of view. By this logic, we would say thatmost syntax/input validation tests are positive tests. Even if you give an invalid
http://www.globaltester.com/http://www.globaltester.com/http://www.globaltester.com/http://www.globaltester.com/


2/7

input, you are expecting a positive result (e.g., an error message) in the hope offinding a situation where the module either gives the wrong error message oractually allows the invalid input. A negative test is, by this logic, more trying toget the module to do something differently than it was designed to do. Forexample, if you are testing a state transition machine and the state transitionsequence is: State 1 -> State 2 -> State 3 -> State 4, then trying to get themodule to go from State 2 to State 4, skipping State 3, is a negative test. So,

negative testing, in this case, is about thinking of how to disrupt the moduleand, by extension, positive testing is examining how well/badly the module doesits task.

Now, in response to this, I would agree that most can see looking at it this wayfrom what the tester hopes to find. Testing pundits often tell testers to look forerror because if you look for success, you will often find success - even whenthere is error. By proxy, if you do not find an error and you have reliable testcases (that latter point is crucial), then a positive test case will show that theapplication did not, in fact, manifest that error. However, showing an error whenit should have done so is an example of a "positive test" by the strict definitionof that term. So in other words:

Positive Testing = (Not showing error when not supposed to) + (Showing errorwhen supposed to)

So if either of the situations in parentheses happens you have a positive test interms of its result - not what the test was hoping to find. The application didwhat it was supposed to do. By that logic:

Negative Testing = (Showing error when not supposed to) + (Not showing errorwhen supposed to)(Usually these situations crop up during boundary testing or cause-effecttesting.) Here if either of the situations in parentheses happens you have anegative test in terms of its result - again, not what the test was hoping to find.The application did what it was not supposed to do.

However, in both cases, these were good results because they showed youwhat the application was doing and you were able to determine if it was workingcorrectly or not. So, by my original definitions, the testing is all about errors andfinding them. It is just how you are looking for those errors that makes the

distinction. (Granted, how you are looking will often dictate what you are hopingto find but since that is the case, it hardly makes sense to make a granddistinction between them.) Now, regarding the point I made above, as a Devil'sAdvocate: "A negative test is more trying to get the module to do somethingdifferently than it was designed to do." We have to realize, I think, that what wecall "negative testing" is often about exercising boundary conditions - and thoseboundaries exist within the context of design. Granted, that can be trying to geta value in to a field that it should not accept. However, a good applicationshould have, during the requirements stage, had provisions for invalid input.Thus really what you are testing here is (a) whether the provisions for invalidinput exist and (b) whether they are working correctly. And, again, that is whythis distiction, for me (between positive and negative), is somewhat banal.

Your negative test can turn into a positive test just be shifting the emphasis ofwhat you are looking for. To get the application to do something it is notdesigned to do could be looked at as accepting invalid input. However, if youfind that the application does accept invalid input and does not, in fact, give awarning, I would agree that is a negative test if it was specified in requirementsthat the application should respond to invalid input. In this case the applicationdid not, but it was not also specified that it should. So, here, by strictrequirements did the application do what it was supposed to do? Technically,yes. If requirements did not specify differently, design was not put in place tohandle the issue. Thus you are not testing something outside the scope ofdesign. Rather, you are testing something that was not designed in the firstplace.


3/7

So, going back to one of the previous points, one thing we can probably allagree on: it entirely depends on how you view a test. But are we saying theresult of the test determines whether it was a positive or negative test? If so,many would disagree with that, indicating that it is the thinking behind the testthat should be positive or negative. In actuality, most experienced testers do notthink in terms of positive or negative, they think in terms of "what can I do to

establish the level of risk?" However, to this point, I would argue that if that istruly how the tester thinks of things then all concepts of positive/negative goright out of the window (as I think they mostly should anyway). Obviously youcould classify the test design in terms of negative or positive, but to some extentthat is irrelevant. However, without getting into that, I am not sure we are sayingthat the result of the test determines positivity or negativity. What I said earlier,relative to my example, was that "in both cases, these were good resultsbecause they showed you what the application was doing and you were able todetermine if it was working correctly or not." If the application was behavingcorrectly or incorrectly, you still determined what the application was actuallydoing and, as such, those are good results. Thus the result tells you about theapplication and that is good (without recourse to terms like positive andnegative). If the result tells you nothing about how the application is functioningthat is, obviously, bad (and, again, this is without recourse to positive ornegative).

We can apply the term "effective" to these types of test cases and we can saythat all test cases, positive or negative, should be effective. But what about theidea of relying on the thinking behind the test? This kind of concept is just a littletoo vague for me because people's thinking can be more or less different, evenon this issue, which can often depend on what people have been taughtregarding these concepts. As I showed, you can transform a postive testmentality into a negative test mentality just by thinking about the results of thetest differently. And if negative testing is just about "disrupting a module" (theDevil's Advocate position), even a positive test can do that if there is a fault.However I am being a little flip because with the notion of the thinking behindthe test, obviously someone here would be talking about intent. The intent is todisrupt the module so as to cause a fault fault and that would constitute anegative test (by the Devil's Advocate position) while a positive test would notbe trying to disrupt the module - even though disruption might occur (again, by

the Devil's Advocate position). The key differentiator is the intent. I could sort ofbuy that but, then again, boundary testing is an attempt to disrupt modulesbecause you are seeing if the system can handle the boundary violation. Thiscan also happen with results. As I said: "Your negative test can turn into apositive test just be shifting the emphasis of what you are looking for." That sortof speaks to the intention of what you are hoping to find but also how you viewthe problem. If the disruption you tried to cause in the module is, in fact,handled by the code then you will get a positive test result - an error messageof some sort.

Now I want to keep on this point because, again, some people state thatnegative testing is about exercising boundary conditions. Some were taughtthat this is not negative testing; rather that this is testing invalid inputs, whichare positive tests - so it depends how you were taught. And figure that aboundary condition, if not handled by the code logic, will potentially severelydisrupt the module - which is the point of negative testing according to someviews of it. However, that is not the intent here according to some. And yetwhile that was not the intent, that might be the result. That is why the distinction,for me, blurs. But here is where the crux of the point is for me: you cangenerlaly forget all about intent of test case design for the moment and look atthe distinction of what the result is in terms of a "positive result" (the applicationshowed me an error when it should have) and a "negative result" (theapplication did not show me an error when it should have). The latter isdefinitely a more negative connotation than the former, regardless of the intentof the tester during design of the test case and that is important to realize


4/7

because sometimes our intentions for tests are changed by the reality of whatexists and what happens as a result of running the tests. So, in the case ofintent for the situation of the application not showing an error when it wassupposed to, this is simply a matter of writing "negative test cases" (if we stickwith the term for a moment) that will generate conditions that should, in turn,generate error messages.

But the point is that the intent of the test case is to see if the application doesnot, in fact, generate that error message. In other words, you are looking for anegative result. But, then again, we can say: "Okay, now I will look that theapplication does generate the error message that it should." Well, in that case,we are really just running the negative test case! Either way the result is that theerror either will or will not show up and thus the result is, at least to someextent, determining the nature of the test case (in terms of negative or positiveconnotation). If the error does not show up, the invalid input might break themodule. So is the breakdown this:

P: Not showing error when not supposed toN: Not showing error when supposed toP: Showing error when supposed toN: Showing error when not supposed toI think the one thing we have to consider is the viewpoint: hinges on the idea of

"negative testing" being looked at as forcing the module to do something it wasnot designed to do. However, if the module was never designed to do the thingyou are trying, then your testing is of an interesting sort because, after all, youknow nothing exists to handle it. So the real question should not be: "Whathappens when I do this?" but rather "Why have we not designed this to handlethis situation?" Let us say that something is designed to handle the "moduledisruption" you are proposed to test. In that case, you are actually positivelytesting the code that handles that situation. To a strict degree, forcing a moduleto do something it was not designed to do suggests that this is something youraverage user can do. In other words, your average user could potentially usethe application in such a fashion that the negative test case you are putting forthcould be emulated by the user. However, if that is the case, design should be inplace to mitigate that problem. And, again, you are then positively testing.

Now, one can argue, "Well, it is possible that the user can try something that

there simply is no way to design around." Okay. But then I ask: "Like what?" Ifthere is no way you can design around it or even design something to watch forthe event, or have the system account for it, how do you write a valid test casefor that? I mean, you can write a test case that breaks the application bydisrupting the module but -- you already knew that was going to happen.However, this is not as cut and dry as I am sure anyone reading this could pointout. After all, in some cases maybe you are not sure that what you are writingas a test case will be disruptive. Ah, but that is the rub. We just defined"negative testing" as trying to disrupt the module. Whether we succeed or not isa different issue (and speaks to the result), but that was the intent. We aretrying to do something that is outside the bounds of design and thus it is not somuch a matter of testing for disruption as it is testing for the effects of thatdisruption. If the effects could be mitigated, that must be some sort of designthat is mitigating them and then you are positively testing that mitigatinginfluence.

As an example, a good test case for a word processer might be: "Turn off thecomputer to simulate a power failure when an unsaved document is present inthe application." Now, the idea here is that you might have some documentsaving feature that automatically kicks in when the application suddenlyterminates, say via a General Protection Fault (GPF). However, strictlyspeaking, powering down the computer is different than a GPF. So here you aretesting to see what happens if the application is shut down via a power-off ofthe PC, which, let us say, the application was not strictly designed to reallyhandle. So my intent is to disrupt the module. However, in this case, since I can


5/7

state the negative condition, I can state a possible design that could account forit. After all: we already know that the document will not be saved becausenothing was designed to account for that. But the crucial point is that if nothingwas designed into the system to account for the power-off of the PC, then whatare you really testing? You are testing that the application does what theapplication does when a power-off occurs. But if nothing is designed to happenone way or the other, then testing for disruption really does you no good. After

all, you know it is going to be disrupted. That is not in question. What is (orshould be) in question is how you can handle that disruption and then test howthat handling works. So let us take those alternate (Devil's Advocate)definitions:

Positive testing is that testing which attempts to show that a given module of anapplication does not do what it is supposed to do.In this case of the power-down test case, we are not positive testing becausewe did not test that the application did not do what it was supposed to do. Theapplication was not "supposed" to do anything because nothing was designedto handle the power-down.

Negative testing is that testing which attempts to show that the module doessomething that it is not supposed to do.In the case of the power-down test case, we are also not negative testing by

this definition because the application, in not saving the document or doinganything (since it was not designed to do anything in the first place), is notdoing something that it is not supposed to do. Again, the application is not"supposed" to do anything since it was not designed to do anything in thissituation.

Now consider my quasi-definition/equation for positive testing that I gaveearlier:

Positive Testing = (Not showing error when not supposed to) + (Showing errorwhen supposed to)I would have to loosen my language a little but, basically, the application wasnot supposed to show an error and, in fact, did not do so in this case. But whatif the application was, in fact, supposed to handle that situation of a power-down? Let us say the developers hooked into the API so that if a shut-down

event was fired off, the application automatically issues an error/warning andthen saves the document in a recovery mode format. Now let us say I test thatand find that the application did not, in fact, save the file. Consider again myquasi-definition/equation for negative testing:

Negative Testing = (Showing error when not supposed to) + (Not showing errorwhen supposed to)In this case I have done negative testing because the application was supposedto issue an error/warning but did not. However, notice, that the test case is thesame exact test case. The intent of my testing was simply to test this aspect ofthe application. The result of the test relative to the stated design is whatdetermines if the test was negative or positive by my definitions. Now, becauseI want to be challenged on this stuff, you could also say: "Yes, but forget thedocument in the word processor. What if the application gets corrupted becauseof the power-off?" Let us say that the corruption is just part of the Windowsenvironment and there is nothing that can be done about it. Is this negativetesting? By the Devil's Advocate definition, strictly it is not, because rememberby that definition: "Negative testing is that testing which attempts to show thatthe module does something that it is not supposed to do." But, in this case, themodule did not do something (become corrupted) that it was not supposed todo. This simply happened as a by-product of a Windows event that cannot behandled. But we did, after all, try to disrupt the module, right? So is it a negativetest or not by the definition of disruption? Incidentally, by my definition, it is not anegative test either. However, what is common in all of what I have said is thatthe power-down test case is an effective test case and this is the case


6/7

regardless of whether you choose to connote it with a "positive" or "negative"qualifier. Since that can be the case, then, for me, the use of the qualifier isirrelevant.

But now let us consider another viewpoint from the Devil's Advocate and onethat I think is pretty good. Consider this example: An application takes mouseclicks as input. The requirement is for one mouse click to be processed at a

time, the user hitting multiple mouse clicks will cause the application to discardanything but the first. Any tester will do the obvious and design a test to hitmultiple mouse clicks. Now the application is designed to discard anything butthe first, so the test could be classified (by my definition) a negative one as theapplication is designed not to process multiple mouse clicks. The negative testis to try to force the application to process more than the first. BUT, I hear yousay, this is an input validation test that tests that the application does discardmutiple mouse clicks, therefore it is a positive test (again, by my definition), andI would then agree, it is a positive test. However, the tester might also design atest that overflows the input buffer with mouse clicks - is that a negative test?.Note, this situation is not covered explicitly in the requirements - and that iscrucial to what I would call negative testing, that very often it is the tester's"what if" analysis that designs negative tests - so, yes, it is a negative test asyou are forcing the application into a situation it may not have been designedand/or coded for - you may not know whether it had or not. The actual result of

the test may be that the application stops accepting any more clicks on its inputbuffer and causes an error message or it may crash.

Now, having said all this, it makes me realize how my point starts to coalescewith the Devil's Advocate. One way that might happen is via the use of the term"error" that has gotten tossed around a lot. My language seemed too restrictivein the sense that when I used the word "error" (as in "showing error when notsupposed to") I did not make it clear enough that I was not necessarily talkingabout an error screen of some sort, but rather an error condition or a failure.With this, my negative testing definition really starts to coalesce with the Devil'sAdvocate's definition ("does something that it is not supposed to do"). I hadsaid: "Negative Testing = (Showing error when not supposed to) + (Not showingerror when supposed to)" and broadening my language more, really what I amsaying is that the application is either showing (doing) something it is notsupposed to (which matches the Devil's Advocate thought) but I was also

saying that the application is not showing (doing) something that it wassupposed to. And to the Devil's Advocate that latter is positive testing. Let merestate the two viewpoints somewhat:

Positive Testing (Jeff):Not doing something it was not supposed to do.Doing something it was supposed to do.

Positive Testing (Devil's Advocate):Not doing what it is supposed to do.

Negative Testing (Jeff):Doing something it was not supposed to do.Not doing something it was supposed to do.

Negative Testing (Devil's Advocate):Doing something that it is not supposed to do.I think I was essentially saying the same thing in terms of negative testing as myhypothetical opponent, just not in terms of positive testing. If you notice, both ofour "negative testings" really contain the same point. On the other hand, relativeto the common Devil's Advocate position, I am having a hard time seeing themajor distinction between positive and negative. The Devil's Advocate's originalconception:

"Positive testing is that testing which attempts to show that a given module of


7/7

an application does NOT do what it is supposed to do. Negative testing is thattesting which attempts to show that the module does something that it is notsupposed to do."

To me, "doing something you are not supposed to" (the Devil's Advocatenegative test) and "not doing something you are supposed to" (the Devil'sAdvocate positive test) are really two sides of the same coin or maybe just two

ways of saying the same thing. So let us say that our requirement is "do notprocess multiple mouse clicks". In that case, "not doing something you aresupposed to" (Devil's Advocate positive test) means, in this case, "processingmultiple mouse clicks". In other words, the application should not processmultiple mouse clicks. If it does, it is doing something it is not supposed to.Likewise, "doing something you are not supposed to do" (Devil's Advocatenegative test) means, in this case, "processing multiple mouse clicks". In otherwords, the application should not process multiple mouse clicks. Either way, it issaying the same thing. So what we are testing for is "application not processingmultiple mouse clicks". It would seem that if the application does processmultiple mouse clicks it is both not doing what it is supposed to do (notprocessing them) and doing something it is not supposed to do (processingthem). The same statement, just made different ways. Now, let me see if thatworks with my definitions.

Again, the lemma is "do not process multiple mouse clicks". If the applicationdoes this then it falls under lemma 1 of my negative test ("Doing something itwas not supposed to do.") If the application does not do this, it falls underlemma 1 of my positive test ("not doing something it was not supposed to do").Even with the mouse click example we have two aspects:

Application designed not to process multiple mouse clicksApplication designed to process only one mouse clickSaying the same thing, and yet a subtle shift in emphasis if you want to go bythe positive and negative distinctions. The difference, however, is also whetheryou are dealing with active design or passive design. In other words, does theapplication actively make sure that only one mouse click is handled (by closingthe buffer) or does it simply only process one click, but allow the buffer to fill upanyway. I like the idea of tying this whole thing in with "mitgating designfactors". I think that we can encapsulate "intent" and "result" (both of which are

important to test casing) by looking more at the efficient and effectivedemarcations. We have to consider result that is part of how you do test caseeffectiveness metrics as well as proactive defect detection metrics. If a test caseis a tautology test then it is not really efficient or effective - but that is basedsolely on the result, not the intent or anything else.

And The Point Is ... ?The point here was simply to show an extended dialogue that I wish moretesters would take it upon themselves to do. Regardless of whether you agreewith some of the conclusions I have put foward here (most notably that"positive" and "negative" is not necessary a very meaningful distinction), at leastyou will hopefully see that it pays to sometimes think out these issues becauseit forces you to better consider what it is you are actually doing when you aretesting and it also helps you think about how to explain it to others.

Documents

Positive and Negative Testing