Jim Holmes Tests Everything January 3, 2012 who's not here for .NET Rocks! speak now. ... an excuse for us to give away stuff. Richard Campbell: Yes. ... like GPS, Wi-Fi

http://www.dotnetrocks.com

Text Transcript of Show #729 (Transcription services provided by PWOP Productions)

Jim Holmes Tests Everything

January 3, 2012 Our Sponsors

Carl Franklin and Richard Campbell

interview experts to bring you insights

into .NET technology and the state of

software development. More than just

a dry interview show, we have fun!

Original Music! Prizes! Check out what

you've been missing!

http://www.telerik.com

http://www.pluralsight-training.net

Transcription by PWOP Productions, http://www.pwop.com Page 2 of 13


January 3, 2012

Lawrence Ryan: .NET Rocks! episode #729, with guest Jim Holmes, recorded live Thursday, December 15, 2011. [Music] Lawrence Ryan: This episode is brought to you by Telerik and by Franklins.Net - Training Developers to Work Smarter and now offering video training on Silverlight 4.0 with Billy Hollis and SharePoint 2010 with Sahil Malik. Order online now at franklins.net. And now here are Carl and Richard. Carl Franklin: Thank you very much. It's Carl and Richard and we're here for .NET Rocks! What are you here for? Richard Campbell: I'm here for .NET Rocks! Are you here for .NET Rocks!? Carl Franklin: I'm here for .NET. Anybody who's not here for .NET Rocks! speak now. Richard Campbell: Or put up your hand because putting up your hand on a radio show is funny. Carl Franklin: All right, we're all here. Oh, my goodness. Richard Campbell: Yes, sir. Carl Franklin: We're having so much fun with the fan club. The .NET Rocks! Fan Club of course is an excuse for us to give away stuff. Richard Campbell: Yes. Carl Franklin: As a matter of fact, it's time to announce the winner of this show's Telerik Ultimate Collection. Richard Campbell: Our very first winner of the year. Carl Franklin: The Telerik Ultimate Collection, of course, is everything Telerik does in one package, 8 grand worth of software. It normally cost $2000 and we're giving one away every show. Richard Campbell: For the whole year. Carl Franklin: All you got to do is join the Fan Club and that's as simple as answering a few questions. Richard Campbell: It is a link right on the home page of www.dotnetrocks.com. Carl Franklin: You can't miss it.

Richard Campbell: Yeah. Carl Franklin: So today's winner is Chris Staley from Louisville, Kentucky. Richard Campbell: Congratulations, Chris. Carl Franklin: I got to get an applause.wav or something like that. Richard Campbell: There you go. Yeah, we really want stadium applause. We want a roar. Carl Franklin: All right, we'll do that. We'll do it again. Ready? Richard Campbell: Okay. Carl Franklin: Chris Staley from Louisville, Kentucky. All right, next time we'll get some real sound effects. We need to find a sponsor so we can get some real sound effects. Richard Campbell: Can we get some professionals around here? Carl Franklin: This is so unprofessional. Well, anyway. Another thing, if you haven't heard that every year we're going to give away some big technology prize, at least $5000 worth of technology, whether it's a computer or a Surface or something with a big honking audio system, I was actually thinking of putting together a system with a subwoofer and like the monitors that we use in the studio. Richard Campbell: The big speakers. Carl Franklin: With a Big Knob. Richard Campbell: Yeah, with the Big Knob, yeah. Carl Franklin: Absolutely. Richard Campbell: Everybody likes the Big Knob. Carl Franklin: Yeah. So you’ve got to join the Fan Club. It's just free stuff constantly. Richard Campbell: Yup. Carl Franklin: So your chances are good right now actually that if you sign up, you're going to win. Richard Campbell: Uh-hmm. Carl Franklin: All right, and with that let's go right to Better Know a Framework. Richard Campbell: All right.



January 3, 2012

[Music] Richard Campbell: First Better Know a Framework of the year, my friend. What have you got? Carl Franklin: All right, here we go. System.Device.Location. Richard Campbell: Oh. Carl Franklin: This is in .NET Framework 4.0. Richard Campbell: Nice. Carl Franklin:` And we've done this before, but it was so much fun and so interesting and so good to know that we brought it back. Richard Campbell: It's yummy. Carl Franklin: So this is where you can access the computer's location using a single API. The location info can come from multiple providers like GPS, Wi-Fi triangulation or cell phone tower triangulation but the System.Device.Location provides that single API to encapsulate all that stuff. Richard Campbell: And it even goes all the way down to stuff like just what locale you plugged in in your control panel. Carl Franklin: Oh, yeah. Richard Campbell: It will always give you something. Carl Franklin: It will give you something. Richard Campbell: Yeah, it's pretty cool. I love that idea, that just, you know, what have we got? Carl Franklin: Yup. So you have a civic address, and a GeoCoordinate, and a GeoPosition which is the location data. GeoPosition(T), right? Richard Campbell: Oh, interesting. Carl Franklin: Yeah. So you can specify a type parameter and then it will return the location data of that type. Richard Campbell: Nice. Carl Franklin: Yeah. So it's very cool. Richard Campbell: It's goodness. Carl Franklin: It's goodness. You're going to need it, especially if you're doing any tablet and mobile stuff, you're going to need that definitely.

System.Device.Location. Richard, who's talking to us? Richard Campbell: Know it, learn it, love it. So I grabbed an email out of the stack to start off our year that will amuse you. Carl Franklin: Okay. Richard Campbell: "Hey, Carl and Richard. I ran into Richard in Frankfurt, Germany airport on November 6." Carl Franklin: Wow. Richard Campbell: Yes, he did. He didn't actually run into me. I don't think there was any physical contact other than a handshake. Carl Franklin: Good. Richard Campbell: "He was on his way to Oredev 2011 and I was on my way to the 2011 IPF World Powerlifting Championship in Pilsen, Czechoslovakia, with a lifter I coach." Carl Franklin: Whoa. Geez. Richard Campbell: Wow, that's cool. Carl Franklin: Yeah. Richard Campbell: "I was surprised to find when I turned around to talk to my lifter to see Richard walking directly behind her. I mentioned that I had attended the last road trip stop in Houston, Texas, back in 2010." Carl Franklin: Yeah, that's right. Richard Campbell: "I had a good little conversation with him in passing about the Geek Out shows you guys have been doing. The podcast you guys did on space a few months back were very interesting. It's staggering to think that some of these high performance planes and spacecraft flew without using the computing power we have today." Carl Franklin: Think about this. The Hoover Dam was built without calculators or computers. Richard Campbell: Yeah. No electronics at all, all done by a slide rule. Carl Franklin: Pencils, papers. Richard Campbell: The Apollo missions were too and the Apollo computer: 8k of memory, baby, 8k. "As I told Richard, I appreciate all the hard work you guys put into .NET Rocks! and the other podcasts you



January 3, 2012

produce. It really helps make the commute through Houston go by faster." Carl Franklin: Oh yeah, we're really working hard. Richard Campbell: That's it. We're slaving away especially when we're talking about space. You know, I’m feeling like we can do a space show every week these days. There's so much going on. It's crazy. Carl Franklin: Oh, I know, I know. Richard Campbell: "I work at Logica in North America on the real-time data management team which has a couple of past .NET Rocks! guests." Carl Franklin: Cool. Richard Campbell: "Our lead dev is J. Sawyer from show #147." Carl Franklin: Oh, yeah. Richard Campbell: "And Michael Stuart from show #24." Carl Franklin: Wow. Richard Campbell: "He's the architect developer on the team." Carl Franklin: Wow. Richard Campbell: "Our team develops complex event processing solutions using Microsoft's StreamInsight. Thanks again, guys." And hey, thanks Tony for saying hi. You know, it sometimes surprises people. It does happen once in a while and I'm happy to chat with anyone. I fly a lot and airports are boring places. It's always good to meet someone who cares about stuff that I'm interested in, too. We had a great conversation although I think your lifter thought we were both nuts because we totally geeked out for the five or six minutes we had walking together before you turned off to your gate and I went down to mine. Carl Franklin: Yeah. Richard Campbell: But the least I can do is send you a mug. Carl Franklin: Really. Richard Campbell: So, if you'd like a mug, write us an email or say hi to us in an airport, email at [email protected] or write a comment on the website at www.dotnetrocks.com.

Carl Franklin: www.pluralsight.com is where you can get 200 free minutes for a 10-day free trial of online developer training, hard core stuff delivered by MVPs and industry experts. They have coverage of iOS, Java, Android, web development, and pretty much anything you can think of on the Microsoft stack. Try Pluralsight today. Subscription plans start at just $29 a month, www.pluralsight.com. Well, this should be an exciting show because our guest is our friend Jim Holmes and Jim's bio is a list. Father. Husband. Geek. Veteran. Around 25 years IT experience. Co-author of “Windows Developer Power Tools.” Coffee Roaster. MVP for C#. Chief Cat Herder of the CodeMash Conference. Richard Campbell: Oh, yeah. Carl Franklin: Diabetic. Runner. Liked 5th grade so much he did it twice. Richard Campbell: Nice. Carl Franklin: One-time setter, middle blocker, and weakside hitter. Blogger (http://FrazzledDad.com). Evangelist for Telerik's Test Studio, an awesome set of tools to help teams deliver better software. And finally, big fan of naps. Hey, Jim. Jim Holmes: Hey, guys. Richard Campbell: So we're starting off our year, welcome to 2012, the first show of the year with a good friend and really this is a remake show. Carl Franklin: Yup. Yeah, we actually did interview you once, but somehow lost it. It happens. Richard Campbell: I blame Carl. Carl Franklin: Yeah. Jim Holmes: Stuff happens. Richard Campbell: Yeah. Not that I've got anywhere to go. I lost a recording with Scott Guthrie so I sort of, I own the biggest mess-up in recording history for us. Carl Franklin: Well, you know, when you do 700 shows, you're bound to lose a couple. Richard Campbell: You're going to drop a couple. Jim Holmes: Yeah. Well, you know it was Vegas. What do you expect? Carl Franklin: No, I can't even blame that. No, it was just sheer stupidity.



January 3, 2012

Richard Campbell: Yeah, stuff happens. Carl Franklin: I pressed the wrong button. Jim Holmes: Yeah, because I've never done that ever before. Richard Campbell: Hey, before we dive into testing, I really want to talk a little bit about CodeMash because we're going. In fact, the conference will be on the week after this show is published. Carl Franklin: Yup. Richard Campbell: But it’s gotten out of hand, hasn't it? Jim Holmes: Yeah, I'm kind of speechless. So six years ago when people asked me how the first CodeMash went, I kind of had to stretch the truth, not stretch the truth. I used marketing positive feedback to say that we had 250 people there. Really, we were around 200. So it was an awesome size, Code Camp size event. This year we're going to have over 1400 people and we sold out in 20 minutes. Richard Campbell: Wow. Carl Franklin: Whoa! Jim Holmes: But what's even cooler is we broke EventBrite. Richard Campbell: Really? Carl Franklin: Wow. Jim Holmes: We had such traffic slamming EventBrite that it overloaded their servers and Brian Prince had to spend like three days working with PayPal records and database transactions and all kinds of -- it was insane. It was just absolutely insane. Carl Franklin: Fabulous. That's awesome. Richard Campbell: I got to ask you this question. Did the Kalahari Resort build an additional wing just to support you guys? Is it the CodeMash Wing? Is that what it is? Jim Holmes: We sure like to think of it that way. Yeah, that's insane as well. So for folks who have been to CodeMash, the previous wing where we were at for venue space was 20,000 square feet. That's a big size. They added on another 100,000 square feet. Richard Campbell: Wow.

Carl Franklin: Wow. Jim Holmes: It's insane. I used to be in the Air Force and the new dining hall is the size of a hangar that we used to fit E3 aircraft, like a 727 aircraft in. Richard Campbell: E3 was a 707, buddy. Jim Holmes: Oh, 707, yeah. What am I saying? Well, it's been a while. Carl Franklin: Richard, how do you know that? Richard Campbell: You know I'm right. Jim Holmes: Yeah, I know you are. Carl Franklin: How do you know that? Jim Holmes: 727 is much smaller, but yes. Richard Campbell: Yeah. Jim Holmes: You know, it's insane. I was there with Jason Follas earlier this week and it's staggering. Carl Franklin: Well, I know the real reason why CodeMash is so popular. Because Richard and I got drunk on a show there once. Richard Campbell: More than once. Jim Holmes: I can neither confirm nor deny that. Carl Franklin: Somebody started passing out bottles of Maker's Mark and then everything got all crazy. Well, anyway. We always have fun when we go there. Richard Campbell: It's a sold out show, so what can we say? Jim Holmes: It is, yeah. Carl Franklin: All right, let's talk testing. Richard Campbell: Yeah. Jim Holmes: I love to talk testing. Carl Franklin: Now, one of the things that we discovered in talking with you the first time was that you didn't become an evangelist and then fall in love with the product. You fell in love with the product, liked it so much that you called up Telerik and said I



January 3, 2012

should be an evangelist for this product because I love it so much. Is that right? Jim Holmes: Yeah, that's exactly kind of how it ran. I've been working with automated testing at the unit and integration level for a long time and then have been, for probably about the last four or five years, doing more and more functional testing so using tools like Visual Studio Webcast, Selenium, WatiN, and you can get a lot done with those tools. There are people doing tremendous things, but I’d also run into a number of the really sharp edges around that stuff and was wondering about, you know, new opportunities and things and then had some conversations with Phil Japikse who had been working for Telerik and found out about Telerik Test Studio and started looking at it and was just really knocked over by the problems it solved that had been kicking my butt for a long time. So yeah, I basically said, "Phil, would you mind passing my resume onto these guys because I'd love to chat with them," you know, and here I am now. Carl Franklin: That's cool. That's great. It's usually not the way it happens. Well, I mean it's great that it does happen that way. Tell us about this product, in particular what it was about it that solved your problems. I guess we should talk about the pain first, huh? Jim Holmes: Yeah. So functional testing at the UI level, regardless of whether you're doing something on the desktop or on the web, is just much more, to my view, it's much more complex, much more brittle than unit or integration tests. There's a lot more moving parts. UI can be really tough. Carl Franklin: Yes, it can. Jim Holmes: Particularly if you're working in a legacy app where, you know, you've got an older web app that's been around for a number of years and the UI is, you know, the structure behind it doesn't look real great. You know, you've got tables within tables within tables, blah, blah, blah. The web app was doing great things. It's making customers happy, but you try to test that and then holy smokes, I've got to deal with this convoluted DOM structure. If you're dealing with -- I mean more and more and more now we're throwing lots of JavaScript at our web apps and that's awesome because we get great things done, but that brings into testing now additional complexity around how are you dealing with the events that JavaScript has on the page. You've got dynamic content were I need to test, write automation, that's going to deal with elements that aren't on the page yet or content that isn't on the page yet. Now you get all these little subtle timing issues that if you aren't very cognizant of exactly what's going on with that page, it's going to kick your butt. And then

there's just overall maintenance of if you are, just like in software, we believe firmly - hopefully - in the ‘don't repeat yourself’ principle. Write dry. Carl Franklin: Right, sure. Jim Holmes: So, you know, that saves our butts on the software side of things. On testing, it's the same thing. So all of those DOM structure things for interacting with elements on a page, if you scatter the definition for those things, how to find those elements, if you scatter those across multiple tests, when your UI changes, that's not an if, right, that's a when, when your UI changes, you're hosed because you're going to have to go through all of these tests and fix all of that stuff. So when I started looking at Test Studio, it solved a whole lot of the things that had been giving me exactly that kind of grief, a much cleaner interaction with the page when you're trying to do things like deal with JavaScript that's on there. We automatically kind of centralize in what we call our element repository all of those locators for the things on the page that you need to interact with so that as you're recording and writing other tests you just automatically refer to that kind of central repository for finding all the things. Carl Franklin: Sort of like creating variables at one level instead of having to nest and find them. Jim Holmes: Yeah, yeah. Yup, very similar to that. So you can do all of this stuff through Selenium and WatiN and the stuff inside Visual Studio, but the thing is you need to spend a whole lot more time in code, say, writing, following what's called Page Object Pattern or -- it's just more and more effort. I saw all this stuff handled in Test Studio and was just really knocked away, blown away by it. So those are some of the specific things. Maintainability around tests is just critical. Two weeks into a testing project and you're all happy because you got all these neat scripts that are playing back and driving the browser around and, "Man, that looks cool, look at this go," and then two months down the road, if you haven't taken some care with what you're doing around your functional test automation regardless of whether it's our tool or somebody else's, you really start running into those problems that I'm talking about. You've got stuff scattered around, your tests are getting longer and longer, it gets harder, and then six months down the road you're like, "God, can I just go back to manual tests please?" Carl Franklin: This portion of .NET Rocks! is brought to you by our good friends at Telerik. Hey, can you ever have too many free tools to complement your development skills? I didn't think so. So our friends at Telerik are giving you now more than 30 free products for application development, automated testing, Agile project management, and content



January 3, 2012

management. And we're talking free-free. Not a trial, not a demo, but free complete products supported by a community of over 440,000 developers at Telerik Forums. From free ASP.NET AJAX, ASP.NET MVC, and Silverlight controls to the free ORM solution and automated testing framework, to free Agile management tools and content management systems, all of these and more are available to you for immediate download at www.telerik.com/freestuff. Most of the free products can be used for commercial purposes and give you access to supplemental support resources such as documentation and forums. Go to www.telerik.com/freestuff now and take full advantage of the available free-of-charge products. And don't forget to thank them for supporting .NET Rocks! Richard Campbell: I can't tell you how many times I've talked to teams where a year in they've abandoned their testing framework entirely and built a new one. Jim Holmes: Yes. Yeah and this happens time and time again. I talk with people in the Ruby domain; they've run into this. I talk with people in .NET; they run into this stuff all the time. Java. These are not technology problems. These are domain problems around testing, and particular around functional testing. Roy Osherove in his awesome book, "The Art of Unit Testing," his opening forward in there is a very frank discussion of a project that he was on or running that the entire project tanked because they really got their tests out of hand and, you know, six to eight months down the road they were spending the vast majority of their time fixing all of their tests and not delivering stuff. Carl Franklin: Right. So tell me about the features that allow for dynamic data to write testing and stuff in dynamic content that isn't there yet. How does that work? Jim Holmes: So first off, let me plug a series that I have going on my blog at www.FrazzledDad.com right now, and it's 31 Days of Testing. I've got a couple of articles on there that speak precisely to this. So when you are trying to write a test around dynamic content, you have to have some kind of a wait. Your script is running very fast and it will just blow through a page unless you tell it, "Hey, wait a minute. I need you to wait here until this next element shows up or until that element is loaded with content," because under AJAX, really that's kind of like the two... Carl Franklin: That's the way it works. Jim Holmes: Yeah. You either don't have an element and that element has to get loaded or you

have the element but the element has no content, right? Carl Franklin: Right, right. Jim Holmes: So there's the idea of implicit waits and explicit waits, and this works the same in our tool or in Selenium. So an implicit wait, you can kind of handle for where the element isn't there yet and you just tell the framework, "Hey, here's kind of the global default. If you can't find what you're looking for, hang out here and wait for 10 seconds and every 500 milliseconds poll the DOM again to see if that element has shown up." So that's a coded step in Selenium. In our tool, as you're recording the test, it's just handled behind the scenes for you so you don't have to consciously deal with that. You still need to understand what's going on, but we take that step of simplifying it for you. Carl Franklin: Okay. Jim Holmes: If you're dealing with the elements there but the content isn't, then you have to do some explicit work. In Selenium, you need to write some additional code making use of some of their infrastructure and support classes where you say, "There's an explicit wait situation on this element. Hang out until you see this content get loaded up and then continue." Carl Franklin: Okay. Jim Holmes: And then in our toolset we've got a very similar notion where if you just add in a wait step that's basically one action in the UI and you say, "Hang out here until you see this content appear and then move on." Carl Franklin: All right. Fair enough. So these are the main things, right, dealing with asynchronous dynamic content, dealing with scripts in content that changes? What is another killer feature that convinced you that this was the tool for you? Jim Holmes: There's a lot of discussion between teams, how do we handle collaboration? Do we have a separate QA group? Do we try to bring folks together? I've never been a fan of the ‘pitch it over the fence to QA’ mindset. So, first off, regardless of whatever teams I've had, whether it's in sports or whatever, in the military or in IT jobs, you just don't segment things out within your team and get the best product out the door. Carl Franklin: Yeah. Jim Holmes: I'm a big fan of having tools where people with certain sets of skills can still get



January 3, 2012

really productive at what they're doing but still foster good communication there. Carl Franklin: Yeah. Jim Holmes: So if you look at the hard core agilists, they're using things like FitNesse and some of the various like Cucumber type, the spec type things where you've got an easy grammar over the top of the tools that are actually doing the work. So those are like the alpha dog teams and they're getting great things done because they absolutely have their tester folks sitting next to their devs. So I look to tools that will support that basically complete coded approach to however you're doing your work. If you're using FitNesse, will the tool fit that? And we can do that, but there are also a huge number of teams where there's maybe not that tight integration or there's still great collaboration but a tester sits off to one side and gets a bit of work done and a dev gets their work done and, you know, maybe they've got just a little bit looser collaboration. So how do you support that? We've got an awesome UI that helps that tester stay very productive. So it's not as intimidating to them as Visual Studio. They can write great tests. They can do very detailed tests and they can write very maintainable tests. When I'm on customer sites, I say the first thing you need to think about is, is this a good test, right? It needs to be granular. You need to make sure that you're testing the right thing because you can get a false sense of confidence through a badly written test. You're not testing the right things. So obviously you've got to take great care with that test to make sure it's the right thing. Carl Franklin: Sure. Jim Holmes: That's priority number one but priority number one-and-a-half is how is this test going to run in six months? Can you make it maintainable? How do you keep things modular? And regardless of whether you're using a Code First framework, Selenium or whatever, or our tool, you have to think hard about keeping those things maintainable. I'm a firm believer that our tool can help keep those tests very maintainable as you're moving forward so that the QA folks, if there's not that tight collaboration, they can still get things done really well. When they need help with doing something like trying to set up calls to a backing API, say for a prerequisite data, you want to set up a backing API that might hit some web services and create users for you. My test isn’t “do I need to create this user?” My test is “Can a user create data in the system?” So I don't want to use the UI to create that guy, so now I can ask a developer to maybe write an API to let me do that so I don't have to do that and get that all that tied in. So that works really well if the dev and the tester are sitting together. It also works well if you've got a tool where they are collaborating through source control,

right? So, I've got my tests, I start to write them, I check it in, in my UI. The dev on the other side of the room or in a different building or the other side of the world polls, gets latest, helps me out with the API, checks things back in, and I can keep going. Richard Campbell: You know, I was looking at Cucumber, which is a Ruby-oriented tool but it is a very kind of a cool idea, but it reminded me a lot of MSpec. I don't know if you've looked at MSpec as well, just this idea of very early specifications as part of the testing process. Jim Holmes: Yeah. So, TDD and BDD everybody is very familiar with or hopefully getting more familiar with. Richard Campbell: They certainly know the acronym. Jim Holmes: Right, so Test Driven Development and Behavior Driven Development. So things like Cucumber and MSpec help with kind of that behavioral type of design and development where can we get an easy grammar that's readable and understandable not just by the developers but by the people that we are building the systems for. Richard Campbell: Right, capturing the domain owner's intent. Jim Holmes: Exactly, exactly, because the XP folks and Agile people are big fans of saying ‘test your documentation,’ Test your documentation for the developers but there's no way that you can hand that off to most business owners or stakeholders and expect them to really understand of validate, "Yeah, this is doing what we expect," but if you're using a tool like Cucumber or MSpec or any number of other tools that are similar to that, then you've got some much more understandable phrasing around “given a regular user has items in their shopping cart. When that user checks out, then they should receive a confirmation email.’ Richard Campbell: Right. Carl Franklin: Right, yup. Richard Campbell: But, you know, that set of specs is a long way away from does this UI do what I expect it to do. Jim Holmes: It is and so, yeah, you're absolutely right because that is showing like a slice, right? Carl Franklin: Yeah.



January 3, 2012

Jim Holmes: So that's the acceptance and, you know, maybe change the grammar around but then yes, so absolutely. For doing something like Cucumber, you've got that grammar text file but then you have to write the fixture behind it that actually drives the UI around and does all of the asserts and then ties things back. So there is a level of hidden goo there that is one or two steps away and you have to be very aware and open about that because that brings its own complexity and, you know, it's a couple of steps away. So guess what? We write bugs in our software. That's another area for us to really screw things up by putting bugs into the test. Whoops. Richard Campbell: Yeah, and we get back to the cycle of I end up with more testing code than production code and it's causing me more time and pain than it gradually becomes, appears to be worth. Jim Holmes: Right, right. Richard Campbell: What do you think a fair ratio is test code to production code? Jim Holmes: So I'm totally going to use the “it depends” cop out. Richard Campbell: Nice. We should just have a little horn that plays. Carl Franklin: Yeah, or sad trombones or sad trumpets. Richard Campbell: Wa, wa, waaaa! Jim Holmes: Honest to God, it does. To my view, it is totally dependent on I guess three factors. One is the size of your project, right? So if I've got a fairly small website, right, that doesn't have a whole lot going on, I'm going to have a very small set of tests for that. So that's one factor, right? So that compared to an online system where you do have a shopping cart and all of these other things. So now all of a sudden I need a whole lot more testing around that just because it's bigger and more complex. The second factor would be, you know, things like risk. So if you're talking HIPAA data or SOX compliance or other regulatory stuff, man, you've got legal risk there so you're going to need much more testing. Richard Campbell: Not that you could eliminate that risk having jumped on this particular hell hole a number of times. You're really creating a document trail that shows reasonable effort to assure security. Carl Franklin: Yeah. Jim Holmes: Yes. Yeah. I mean we can't, what is it, you can't disprove negatives, right, but you can show we've got these things that we understand

were important and here's what we've done plus we've done exploratory testing around this following kind of these patterns because I think that's also extraordinarily important to have involved in the mix as well. Automation is 100%. But yeah, you're right. So it's giving you some exposure and awareness of what your risk is. And then I'd say the third factor that contributes to how much test, you know, where do you stop with tests is, how good is your team. There have been teams where I've been very comfortable with minimal testing because they collaborate, they talk, they write great code, and we can be very focused about where we spend our automation time. And there have been other teams where it's like, man, you know, cut your projected velocity in half because we're going to have to be writing a whole lot more tests and take it from there. Carl Franklin: At Franklins.Net right now, you can get a DVD with over 11 hours of Billy Hollis on Silverlight 4 or 14 hours of Sahil Malik on SharePoint 2010 each for only $695. Order online at www.franklins.net. Are you looking to change jobs? Infusion Development has offices in New York City, Toronto, London, Dubai, and Poland. Infusion has hired a whole handful of happy .NET Rocks! listeners. Contact me for an introduction at [email protected]. Jim, do you think that the test-first philosophy also applies to functional testing as much as it does to unit testing? Jim Holmes: I do. I really do. So there's a whole another offshoot of the *DD which is Acceptance Test-Driven Development and this would really be where you'd start to use things like... Carl Franklin: How many kinds of tests can we drive development with? I mean what? You write your unit tests then your functional tests then you, you know, and then you start writing your code? Jim Holmes: I think that the acceptance test is this really ATDD. So alphabet soup, other variants of this. The idea is that you write a spec or you can think of it as -- so I didn't invent this term but there's a great phrase called outside-in development. So you write that outside layer of here's the acceptance criteria that we need for this user story or for this one work item, right, because you want to stay granular. Carl Franklin: Yeah. Jim Holmes: So here's what this needs to do. When we're done, this thing needs to pass and you get that outside, and then you go back and you start from the inside and work back out. So then you go down and you start doing, you know, more of what



January 3, 2012

we understand in the .NET world as the TDD approach, right? So unit tests and some integration tests, and you build out the functionality until that spec passes and then you've got awesome coverage. You've got that functional test that validates yea and verily. The system does for this slice what we expected it and you've got the unit tests and maybe some integration tests in there that all of the internals are working as we need it to. Carl Franklin: Okay. Jim Holmes: So you know there's a lot of moving parts in there. Some people will freak out when you start talking about that much testing but if you look to a number of folks that are writing about this stuff, it keeps the systems extraordinarily lean. You're very focused on delivering great value because you're doing things in small chunks and you're being very precise about what it is you're building and how the system is supposed to react. Sorry, go ahead. Carl Franklin: So it just seems weird to me that you would write a functional test to test UI, for example, before the UI is even written. Jim Holmes: Yeah. So you can't -- depending on how your team is set up because all of this is very context specific, you know, how is your team work, but at least to that thing of, look, here's your work item that we're working on today and maybe tomorrow. Here are the acceptance criteria and even if it's just scratched on a 3 x 5 card, at least you've got a clear notion of this is what this feature or this item needs to specifically do. Carl Franklin: Okay. Jim Holmes: And you can start writing that at the UI level, but you're going to stub things out, right, because you don't have necessarily all of your element locators to interact with. Carl Franklin: It sounds like you're saying that you do them in parallel. Jim Holmes: Yeah, because you get as much done as you can while you're having constant conversations between the testers, the PM, stakeholder, whatever, and the dev and hopefully they're in the same room having these kinds of conversations. If not, then you spend a lot of time on Skype or whatever, but that's where you get the best actions going on is with that level of communication. Richard Campbell: It still gets into the real challenge of I'm a big believer in setting good goal posts for a project, like when will that be done, recognizing that code is never done.

Jim Holmes: Right. Carl Franklin: [laughter} Richard Campbell: Yeah. At some point you stop, right? Jim Holmes: Yeah. Richard Campbell: No projects are ever finished, only cancelled. Carl Franklin: Yeah. Richard Campbell: Yeah, whether the code's still running that's another question entirely. But this idea that we could actually build automated tests, that you could set the goal post so clearly in the ground, that the moment the thing goes past, alarms can go off, it's done. Jim Holmes: You know, that's really huge. So to me there's a couple things that are critical around making that happen and the first is being very granular with your work item size so that we stay working small little chunks that we can roll out very quickly to the customer, and then the notion of those good specs and then also everything on the backlog is, you know, the customer needs to be able to own that, reprioritize it, and they get to decide when we're done. Yes, we roll out one little small chunk and they say, "Man, you know what? I've hit that 80%. I don't need to go the other 20% which is going to cost me a huge amount more money." Richard Campbell: Right. Jim Holmes: We've got clear goal posts, like you said. It's a great metaphor. We've got the tests documented or the tests running that show this works as expected. Let's go find another project to work on. Richard Campbell: Well, yeah, and an interesting point that I was not just being factious when I said it only ever gets canceled. It's just like typically we keep adding features until the customer satisfaction hits a point where that's enough. Jim Holmes: Yeah. Richard Campbell: The cost is no longer justified, whether you can do that. My concern with that, especially in internal development, is like this. There's never a celebration of success. Carl Franklin: Yeah. Richard Campbell: There's always a demand to stop.



January 3, 2012

Jim Holmes: Yeah. Richard Campbell: Don't we have something else to work on now because this is, you know, that last rev, the last set of changes just doesn't seem all that important. Nobody seems to care. Move on. As opposed to actually setting stakes in the ground so that you can have a party. Carl Franklin: It's kind of like updates to your Facebook iPhone app. Richard Campbell: [laughter] Carl Franklin: No, I'm just kidding. Richard Campbell: Not that you're bitter or anything. Carl Franklin: Oh I'm sorry. Yeah just, you know, the latest one I upgraded to and it has the added feature of when you press any button, it just hangs for five minutes. Jim Holmes: Oh, man. Oh, that hurts. Richard Campbell: Yeah. Carl Franklin: Yeah. I'm so glad I upgraded that. I wish I could roll it back, you know? I want my money back. Give me that last version. Jim Holmes: Oh, man. Yeah, that hurts. Richard Campbell: Where do you put stuff like load testing into the equation, Jim? Jim Holmes: So again, those small sites generally are not going to need that. Richard Campbell: Yeah. Jim Holmes: Load and performance and stress testing, those terms - I'm telling you this - those terms really get mixed and muddled around a lot. Richard Campbell: Sure. Jim Holmes: So to me almost every app should have some kind of a performance test, and that's simply just I have one test that runs a vertical slice through a couple important parts to the system and I just understand how long it takes me to run that one thing, one user, one action. I know how long it takes. I've got a regression trip wire, because then I can understand if I really screwed something up. Carl Franklin: Yup.

Jim Holmes: You should get that in place to start because it's not tough to do. There are any number of tools. We rolled out performance testing in our last release and it's really easy to use. If you're on other platforms, there are all kinds of tools that will let you do that at a pretty cost effective fashion. Get that kind of stuff in early because it's not hard and it gets you great information, but when you step up to like load testing where I need to throw 3,000 users at my website and make sure that my shopping cart still works so that I know I can earn money and get revenue in, revenue matters, that's a horse of a different color and you need to do a lot of planning ahead of time to figure out is this going to be cost effective? Do I actually need to get that wrapped into our process? And if you do, you need to get all of that stuff invested very early on because it's not something that is easy to do after the horses run out of the barn, to make a bad metaphor there. Richard Campbell: Well, and throwing my IT hat on there, often I have a Service Level Agreement that talks about what the page performance and so forth has got to be and I can start -- I wish there was a great tool that went from SLA document to setup tests. The nearest I can tell it's a couple of guys who... Carl Franklin: It's called a programmer. Richard Campbell: Yeah and generally, you know, the guys that are good at that sort of stuff, not fun guys to hang around with either. Like, they're really serious. They count the numbers really tightly; how long does it take to fill in this form given that a human has perfect entry and no hesitation and then how long does it take to get stored, yeah. The interesting thing is there are persons out there that are well-built to do that, these SLA prosecutors, because that's their job, right? I've had versions, certainly versions delayed. An app that has got three or four versions out, there's great resistance to shipping a new version from the operation side. Jim Holmes: Yeah. Richard Campbell: It's like we're currently meeting our SLA and you're going to jeopardize that by pushing a new version. Prove to me you can make these benchmarks. Jim Holmes: Yeah and that goes back to revenue coming in the door, right? Operations is comfy. The system is stable. They know what its performance is. Earning money is okay, it really is. Richard Campbell: Yup. Jim Holmes: And we talked about risk earlier. Can you show me a clear understanding what



January 3, 2012

the risk of this new roll out is? And if you haven't done your load testing work and gotten all that documented, you don't have much of a foot to stand on. Richard Campbell: Well, and on the performance side, I'm talking about Strangeloop here, in terms of performance testing, we actually have a set of benchmarks for every single feature and how they currently perform in the previous version and built into the testing sequence is testing all the times on all those things and if the times get faster, which does happen, people have worked on that sort of stuff, you know, there's a party, it gets flagged, and when it gets slower it also gets flagged. What it really means is that we have to consciously decide we know this is slower because we've added these things. We don't think we can make it any faster than that. We’ll accept that. Not just this default acceptance of that stuff is now slower, right down at the individual feature level. Jim Holmes: Right. Carl Franklin: Yeah. Jim Holmes: So much of testing at the broad level. . . you know, I wear a dev hat, I wear a PM hat, I wear tester hat. As a tester I can't guarantee you quality. I can't. What I can do is help you understand the risk of deploying this software and that is absolutely what my responsibility is to business. I'm not here to be the QA Nazi that says, "No, we're not releasing this." Carl Franklin: Right. Somebody else makes that decision. You just give the numbers. Jim Holmes: Absolutely. You need to give them kind of your due diligence and, "Look, here's what we as a team have done for you. Here's how we feel about it. Here's some good exposure, metrics around performance, past testing history," blah, blah, blah. It's about letting the business know what the risk is to move forward. Carl Franklin: So shall we give away one of these Telerik Testing Tool Suite? Suites? Jim Holmes: That would be awesome. Absolutely, sure you can. Carl Franklin: All right. So what we'll do is we'll pick a name from our fan club and we'll announce that on the next show. Jim Holmes: Oh, that would be very cool. Yup, so I'd be happy to give away a license for Test Studio. So that comes both with the standalone version and then also everything -- most of what I've

talked about, you can definitely do straight with inside Visual Studio as well. Carl Franklin: Okay and just tell us briefly what the options are for purchasing. Jim Holmes: So Telerik Test Studio comes in -- so we have a couple different options through Telerik. If you own the Telerik Ultimate Collection( so that's like all of the, you know, 10 billion awesome control suites and all of the Just series for productivity), if you own that Ultimate Collection, you already own a license for Test Studio Express which is our Visual Studio plug-in. Carl Franklin: Yup. Jim Holmes: So you get the great framework. You can record, test, play them back, do a lot of great stuff through there; all of that maintainability stuff that I'm so passionate about works right in there. If you buy Test Studio itself, which is $2,499, you get standalone UI which brings in performance testing, manual testing, a number of other things, and you also get that Visual Studio feature. Carl Franklin: Now, isn’t the Ultimate Collection only $2000? Jim Holmes: You know what? I don't know what the price on that is. Richard Campbell: Yup. Carl Franklin: Yeah. I think it's $1,999. Richard Campbell: Yeah. Carl Franklin: So the Ultimate Collection is less than the Test Suite by itself. Jim Holmes: Correct. Carl Franklin: Hmm. So that's very cool. Well, all right. So what we'll do is like I said before. We'll pick another winner for today's show and we'll announce the winner of that Test Suite in the next show from the Fan Club, which by the way you have to be a fan to win stuff on .NET Rocks!. Go to www.dotnetrocks.com/fanpage.aspx and sign up. Well, Jim, this has been very cool. Jim Holmes: I enjoyed chatting with you guys again. Carl Franklin: It's great that you're so passionate about what you're evangelizing. That's just what it's all about.



January 3, 2012

Jim Holmes: Yeah, you know, I've wandered around, done a whole bunch of different things, and testing is something that I've been passionate about for a long time. I love the tool set that I'm working with, but even if you don't use our stuff, there are many important things that I feel the need to talk with people about because we're getting so many things wrong that we shouldn't be. It's through some ignorance and people not understanding things or not keeping up or just not having had a guiding light to show the way, I'm not some prophet holding a guiding light out there, but I love being able to spread the word and say, "Man, I screwed this up many times. Learn from my pain. Here's a better way to do this." If you want to use our tool, great, I love that, but be successful in what you do. Listen to me. Listen to some of thes No, I'm just kidding e other folks because, man, we can help you out and there are just so many smart people in the automation industry that, God, my head just explodes learning all this cool stuff. Yes, I'm obviously a little excited and passionate about it. Carl Franklin: Yeah, that's great. Well, Jim, thanks again. Jim Holmes: Thank you, guys. Carl Franklin: It's been wonderful talking to you. And we'll talk to you next time on .NET Rocks! Thanks for listening and remember www.pluralsight.com is where you can get 200 free minutes of developer training online, www.pluralsight.com. [Music] Carl Franklin: .NET Rocks! is recorded and produced by PWOP Productions, providing professional audio, audio mastering, video, post production, and podcasting services, online at www.pwop.com. .NET Rocks! is a production of Franklins.Net, training developers to work smarter and offering custom onsite classes in Microsoft development technology with expert developers, online at www.franklins.net. For more .NET Rocks! episodes and to subscribe to the podcast feeds, go to our website at www.dotnetrocks.com.

Documents

Jim Holmes Tests Everything January 3, 2012 who's not here for .NET Rocks! speak now. ... an excuse for us to give away stuff. Richard Campbell: Yes. ... like GPS, Wi-Fi