Jim Weirich put up a potential Ruby test framework that steals the Given/When/Then syntax from Cucumber. I spent some time noodling along similar lines some months back but never got very far with it. Interestingly, Jim’s proposal and my idea seem to be a little bit different, so I thought I’d write up what I was trying to do, and maybe actually try and make it real.

I was thinking about a particular problem that’s easiest to see in a Rails controller test. Using Shoulda, and trying to use a nested context, one assertion per test style, I was trying to test admin and regular user behavior in a particular controller. I started with something like this:

  context "GET edit" do

    setup do
      @proj1 = Project.make(:name => "Project Runway")
      get :edit, :id => @proj1.to_param
    end

    should "have something" { assert something here}
  end

Fine and dandy, but when I want to split this into admin and regular users, what I really want to do is this, bear with the slight pseduo-code:

  context "GET edit" do

    setup do
      @proj1 = Project.make(:name => "Project Runway")
      get :edit, :id => @proj1.to_param
    end

    context "with a normal user" do
      setup { @user = login! }

      should "have something" { assert something here}  
    end

    context "with an admin user" do 
      setup { @user = admin! }

      should "have something adminy" { assert something here}
    end
  end

Unfortunately this doesn’t work, since the get is called before the subordinate setup clauses, the user doesn’t take effect until after the controller call has been made. The workaround is to move the get call into the subordinate setup clauses, which is hardly the end of the world or anything like that but which did bug me a little.

Thinking about this, it occurred to me that the user set up is basically Given, while the get call is basically a When, and what I want to do is insert a new given, while keeping the same when.

I never did quite nail the syntax, but I was hoping for something like this

  Scenario "GET edit" do  
    Given { @proj1 = Project.make(:name => "Project Runway") }
    When { get :edit, :id => @proj1.to_param }

    Scenario "For a normal user" do
      Given { @user = login! }
      Then { assert something here}
    end

    Scenario "For an admin user" do
      Given { @user = admin! }
      Then { assert something here}
    end
  end

The semantics of this is that any Then clause would trigger a test, first all the Given clauses would be run for all nested scenarios, then all the When clauses, followed by the Then block, which would need to return true for the test to pass.

I never got around to implementing it, in part for lack of time, and in part doubting whether this occasional case would be worth the extra effort. As much as I like how the Given/When/Then structure reads here, I’m not sure that it would be clear to a reader of this test that the subordinate Given clauses would be executed before the When clause. Though I guess that would come after some time with the framework.

Thoughts? Am I crazy?

The Testing Practices Interview series is back. Our interviewee is Mike Gunderloy, publisher of A Fresh Cup and member of the Rails Documentation Team.

Mike has recently released Rails Rescue Handbook, a guide to taking over Rails projects that have been abandoned or are otherwise new to you. It’s a DRM-free PDF file, available for $9.95, and it’s full of useful tips and advice, highly recommended.

So, here’s Mike.

How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?

To some extent I’ve been testing for 30 years, ever since I got started programming. I don’t see a strong distinction in purpose between manual testing, debugging, and automated testing: they’re all ways to make sure your code does what you think it does.

As far as automated testing, though, my first exposure was during some Microsoft contracts 10 years ago or thereabouts. But on those big projects, automated tests were maintained by the testers, not by the developers; completely different group of people and tools. I did dabble a bit with tools like TestDriven.NET, but ultimately, it was moving over to Ruby/Rails that got me into a truly test-infected environment.

So I’ve pretty much “always” known that testing was valuable, and even seen the benefits of automated testing for quite a long while. It’s been an evolutionary step from there to developer-written automated tests and TDD – which I am not religious about. I use TDD when it suits me, and skip it when it doesn’t.

What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?

Probably 90% of the tests I write these days are functional and unit tests, and the other 10% integration tests. As a subcontractor to many different teams, I’ve had the chance to experiment with a great many tools. The exposure has left me unconvinced that there’s a good ROI on either view testing or Cucumber-style BDD, though I recognize that reasonable people differ on this (and that different teams or different projects might have left me with a different impression).

When I have the luxury of picking my own tools these days, I find myself using Test::Unit, Shoulda, Mocha, and Object Daddy most often, with a side of parallel_test when the test suite grows. I’ve played with a couple of different CI servers as well, and at the moment I’m leaning towards Cerberus.

As far as process goes it’s a mix of TDD and more traditional top-down decomposition development with tests as I go; some of this just reflects my age in the industry, I’m sure. I do lots of little code spikes via git branching, and those usually don’t have tests involved; then I’ll come back to the branch I’m working on when I know where I’m going and write the tests and code together.

What’s the most interesting thing you’ve discovered about testing recently?

I’ve been spending a fair amount of time doing code reviews lately. It astonishes me how many teams – even good teams writing good code – have little to no test coverage. The fact that the code is good indicates that testing is not 100% essential – the fact that it has holes shows how it would help. And it’s also evident that the wider Rails community isn’t as test-infected as we like to think it is.

Is there a tool you wish you had for testing that you don’t think currently exists?

I wouldn’t mind finding a good open source record-and-playback tool to use for browser-based integration and maybe even view testing. That used to be pretty useful, back in the day. For all I know one exists and I haven’t gone looking for it. I’d also like something that could do an automatic “lint” looking for cross-browser issues.

What advice would you give somebody looking to write more effective tests?

0) Buy Noel’s book. 1) Write tests. Like anything else, you improve if you do more of it. 2) Find a project you like and look at their tests to see how they did it. If you find patterns you can steal, go for it.

Testing was a popular topic at RailsConf this year, with a lot of sessions on various testing tools and a lot of prominent interest in testing process.

Here’s a quick look at the sessions that were interesting from a testing perspective. And also my session, which was from a testing perspective as well. Where applicable, links go to the presentation slides.

Testing, Design, and Refactoring, Jim Weirich and Joe O’Brien.

This was a tutorial session that I caught a tiny piece of. Weirich has a well-deserved reputation for running great training sessions, and this one seemed effective.

Don’t Mock Yourself Out, David Chelimsky

Saw this one. Chelimsky, who is the lead developer of RSpec, gave a nice overview of how to use mock objects, and even better, how to use them well.

He also introduced a couple of new tools. One was stub_chain, a new feature of the RSpec mock framework. The use case here is when you had to create a series of mock objects to cover a long chain of objects. In the old way, you needed to create a bunch of intermediate objects that you didn’t much care about:

  member = stub_model(User)
  friends = stub()
  friend = stub_model(User)
  User.stub(:find).and_return(member)
  member.stub(:friends).and_return(friends)
  friends.stub(:favorite).and_return(friend)

The new way, with stub_chain, is much more direct.

  friend = stub_model(User)
  User.stub_chain(:find, :friends, :favorite).and_return(friend)

In both cases User.find(1).friends.favorite will return friend.

Another new tool is Stubble, an API for easily mocking ActiveRecord objects. Right now it works with RSpec, but Chelimsky hopes that it will work with other frameworks soon. Within a stubbing block, the ActiveRecord class specified has find, save, and the like stubbed out so that it will not need to contact the database.

  stubbing(Registration) do
    post 'create'
    response.should redirect_to(registrations_path)
  end

These tools, along with some other comments at the conference, lead me to think I need to give RSpec another serious try.

In Praise of Non-Fixtured Data, Kevin Barnes

Missed this one.

JavaScript Testing In Rails: Fast, Headless, In-Browser. Pick Any Three. Larry Karnowski, Jason Rudolph

Only caught the tail end of this one. The tool being discussed here is Blue Ridge, which seems to be a composite tool making it easy to use some JavaScript test frameworks together.

Below and Beneath TDD: Test Last Development and Other Real-World Test Patterns Presentation, by Me

This was my talk. It was pretty process heavy and code-light. I’m going to try to work the substance of it into the book sometime over the next few weeks. Anyway, people came, they said nice things, and overall I was pleased with how it went.

Quality Code With Cucumber, Aslak Hellesøy

I didn’t make it to this session, although I heard very good things about it from other attendees (though I did get to briefly meet Aslak during the conference). Cucumber was all over RailsConf, mentioned in several different sessions, it clearly has a lot of momentum as a useful tool.

Using metric_fu to make your Rails Code Better, Jake Scruggs

I only realized when looking at the slides that this was a descendent of the talk that Scruggs gave at last years WindyCityRails conference. It’s a good talk, and metric_fu is a useful tool.

Working Effectively With Legacy Rails Code, Pat Maddox, BJ Clark

Very well done session that had some suggestions that went beyond what I currently have in the book, although they started with similar advice. A couple of their suggestions include:

• Silo code by making it a webservice.
• Refactor back to what Rails already does.
• Make everything a framework. Write micro frameworks.
• Use “nice” metaprogramming
• Use existing frameworks, then maintenance is somebody else’s problem.
• Look for ways to modify code behavior without changing API, like new parameters with default values
• Use selenium recordings as a last-ditch way of testing.

Robert Martin

This link is to the video of the talk. This talk is included in the testing roundup because of Martin’s full-throated support of test-driven development as a professional duty. I very much enjoyed this talk, though I’ll freely admit that it was very much a preaching to the choir kind of a thing. (Not only did Martin strongly endorse TDD, but his diagnosis of what killed Smalltalk was very similar to rants I’ve been doing for years. You’ll just have to take my word for that.)

Testing BoF session

After some dancing back and forth over room assignments, I settled in with a couple of dozen other people for a discussion of testing that largely centered around the best way to get started and the best way to get unstuck. I had fun, but then I got to shoot my mouth off for an hour, so that may largely be redundant.

Webrat: Rails Acceptance Testing Evolved, Bryan Helmkamp

Solid session on Webrat that went far enough beyond beginner material that I felt that I learned some useful stuff about using Webrat in either Cucumber or Rails integration tests.

Overall, a very strong and helpful conference, even with just testing sessions counted. I’ll try and get another post up with more personal comments later this week.

Questions, I get questions. Well, sometimes I do. Here’s one. And sorry, this should have been posted here a couple of weeks ago.

[In my project] Only admin users are allowed to add users or inactivate them which means among other things that the user must be an admin and logged in.

I struggled for a while with trying to make a unit test work with this and had issues with the fact that there’s no active session while running unit tests. That further revealed that having this logic in the model is problematic too. People posted ways to make this work, but it appears to be frowned upon. Eventually, I just removed this from the model and placed it in the controller and correspondingly moved the test to the users_controller_test.rb.

So, the correct place for this test is the functional test for users from a testing/functionality philosophical standpoint?

Short answer:

I agree with your conclusion. Logic having to do with authentication and access should be tested in the controller test.

Longer answer:

As you correctly note, sessions are not available in the model test environment, which in my experience is a hint and a half from the Rails core team that they really don’t want you doing that. Rails core is never particularly shy about denying or limiting access to things they don’t want you to be able to do easily. That said, it’s not that hard to fake, see the section in the book for testing helpers for an example of how to get session features in a unit test environment.

From a philosophical standpoint, you want to split the access from what is actually the logic in the model. In other words, whether you need to be a logged in admin to get to the add users page is properly a controller test, so is the idea that if you check this box, the the user’s active status changes. However, if changing the users active status triggers other logic in the model, like some kind of activity history or some such, then there should be a model method with a name like “inactivate” and that method should be tested in the model tests.

Crossposting alert: I have a post about test structure and nested contexts up at the Pathfinder Agile Ajax blog. Enjoy.

I heard from Lulu.com support yesterday, and apparently they don’t, in fact, support the kind of updating that I thought they did. Which is my fault, I guess, although the comments in their support forum before I set up the book there led me to believe otherwise.

I’ll be setting up a registration system on this site in the next couple of days where people who purchased the book on Lulu will be able to get updates easily. Long term, this means I’ll move the purchase site for the book, but I don’t have any place to put it at the moment, so I’m stuck with this rather unwieldy process for the interim. Which is irritating for me and for you.

I’ll keep this short and sweet.

Rails Test Prescriptions is now on sale.

The Lulu URL is http://www.lulu.com/content/e-book/rails_test_prescriptions/6418439. Price is $9.00. Please let me know what your purchase experience is like—and keep the Lulu receipt. I might use it for update registration in the future.

I hope you like what’s there and what will be coming. Please email at railsprescriptions at gmail dot com, or use the Get Satisfaction feedback link, or comment here. Your feedback will make a huge difference in how this project proceeds.

Also, I feel a little silly mentioning this, but I also feel silly not mentioning it. If, for some reason, you feel like supporting this project beyond purchasing a copy—like, you want to distribute the book throughout your organization, there’s a PayPal donate link on this site. Funds will go toward improving the book and this site.

Thanks for all your support and kind words over the last few months.

Once upon a time, I worked for a large telecommunications company. In fact, I just wrote a little bit about it over on the Pathfinder Agile Ajax blog. Which reminded me that I had been meaning to talk about how my team did testing on my primary project there. This is in the spirit of a real-world process report, or doing the best you can under less-than-perfect conditions for test-driven development.

Here’s a brief, necessarily somewhat vague, description of the project. It was a Java Swing UI for configuring an elaborate network. The input was several screens worth of information about the topology of the various devices. The output was dozens, if not hundreds, of text configurations for the devices in the network. There was also a validation step, where the configuration would be tested against rules for legal configurations that were frankly quite Byzantine.

Anyway, at the point in time we’re talking about here, the main part of the application had a 5-tier architecture (developed before I got there, and not from a test-driven process) such that creating object clusters for unit tests was prohibitively irritating. With significant investment in creating mock structures, we were able to reduce that to merely very irritating, but even so, large parts of the system, especially in the GUI, were basically opaque to unit testing without a total overhaul.

The output tier was a little better. Output was managed via Velocity templates, and the context objects behind them were developed test-first, and were separated enough from the rest of the system such that unit tests for the logic were feasible there. However, the actual Velocity output wasn’t easy to test directly—Velocity doesn’t have an easy mechanism for testing partial output in a unit test structure.

Two more points, and then we’ll get to the plan.

• You know how I say that tests are a great device for testing that the program does what the developer thinks it does, but not all that great for testing that it’s doing what it’s supposed to do? Big factor here. Since the output of the program was a hardware configuration file, there were an infinite number of ways that the program could avoid error while still producing flawed output.

• While my immediate management was never anything other than supportive, the stance of upper management was a little weird. They were happy to appropriate the word “agile” to describe their ideal process, but were very down on developers actually doing testing of any stripe. This led to some friction in our system—for example, it took a very long time for us to get a suitably powered continuous integration server. On the plus side, the actual team of developers was super-great.

So… what would you do?

Well, here’s what we did. I’m not prepared to defend this as optimal, but it did mostly seem to work.

• On the existing code, especially the GUI, we largely punted on testing. The effort involved, what with the legacy architecture and all, made setting up unit tests prohibitively expensive. We still tried in the somewhat less encumbered data layers, but a wide swath of functionally was basically trapped in static classes with a lot of interdependencies.

• The Velocity contexts, which were more isolated from the rest of the code, came to have much of the output logic. This code, being somewhat newer, actually tended to be written with tests. We also felt that, given the choice, it was more important to nail down the output.

• The actual Velocity output was complicated to test, so we built two tools to manage it. The first was a golden output test harness. We created something like two dozen sample systems that covered as much of the problem space as we could. The output was verified by our hardware experts, and stored in our source tree. The test harness would run the new code and compare the output to the golden versions. The harness (thanks to an excellent job from a fellow team member) ran Beyond Compare on the two directories, making it super-easy to see the changes, approve them, and update the golden version. The bottleneck was not, as you might expect, changing the files themselves. Rather, it was the super-slow source control in use. An update to a common header could easily change hundreds of gold output files. Changing the files would take a minute, updating the source control could take two hours. Really. (Running the tests themselves took about ten minutes—too long to run in a tight loop, but something you could run before checking back to source control.) Despite the speed issues, this turned out to be a very useful way for us to keep confident in the validity of our output, and also an easy way for us to have a context for discussion with our hardware experts.

• Which left a separate problem—ensuring that the gold output tests actually covered all the possibilities. Velocity doesn’t have an actual coverage tool, or at least didn’t at the time. So… we wrote a Jester-style mutation tool that randomly changed lines in the Velocity template and checked for broken tests. This one did take a while to run, so we only did it sporadically, but it was useful as a sanity check from time to time.

Again, clearly not optimal, but it did let us get as much of the verification and confidence benefits of testing within the constraints of the system and the environment.

And that’s it for this week’s episode of “silly testing things I’ve done in the past.” Thanks for listening.

I’m very excited to have Chad Fowler as the latest participant in the Testing Practice interview series.

Chad Fowler is an internationally known software developer, trainer, manager, speaker, and musician. Over the past decade he has worked with some of the world’s largest companies and most admired software developers. He loves to program computers and, as part of his role as CTO of InfoEther, Inc., spends much of his time solving hard problems for customers in the Ruby language. He is co-organizer of RubyConf, and RailsConf and author or co-author of a number of popular software books.

And so, let’s hear from Chad…

How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?

For me it was when I was exposed to TDD for the first time. I had been practicing the typical “guru reads the output” style of testing, primarily in Java at the time. That means every class typically had a main() method at the bottom which would exercise my code and print a bunch of hardly decipherable junk to the screen. As I was developing, I knew what that junk meant. Nobody else ever did. Two days later, neither did I. But the main() methods remained. Because, hey, those were the tests.

I think the way most people worked in an environment like that is that if they needed to change anything in the production code, they would hack the existing “tests” in the main() method such that they could understand what they meant (since this stuff was embedded in production code, you didn’t typically venture out of main() for fear of polluting the namespace). So it was more of a scratchpad than a test.

At the same time, I was doing a lot of mentoring of junior developers on how to do object oriented programming. Specifically, I was trying to help a large group of developers stop generating unmaintainable spaghetti Java. I started using a technique that I call Error Driven Development. It was very much like TDD, which I had not yet heard of. You start in the area of the application you’re trying to implement. Maybe it’s a controller in an MVC setup, for example. And you type in the code you wish already existed as an API. You express your intent as clearly and succinctly as possible within the scope of what would be technically possible in this magical but not yet existing API (paying attention to which data needs to be passed around as parameters, which imaginary objects would best play the role of owning which function, etc.). Then you try to compile and/or run your code and follow the error stream until everything works.

With this kind of development, you get to always work at the level of abstraction you’re interested in while you’re developing. When I’m in a controller, I don’t care about SQL. When I’m in a view, I don’t care how business logic is implemented. To make everything run, of course, I have to walk down through the layers of abstraction and repeat this process—-imagining the perfect API and just using it. Eventually I’m done. It’s motivating and makes better code.

So, the question was about testing. This Error Driven Development technique was just a lame version of how good TDD works. TDD gives you the extra advantage of a nice framework with assertions and reports. So the first time I saw TDD, I was hooked. It was a better version of what I’d been working toward in my quest to write and help others write better code. It just happened to also create tests as a side effect.

What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?

I use whatever testing framework and tools the team I’m working with uses. I’ve done everything from out-of-the-box test/unit to RSpec. Given a choice these days, I’d probably depend on Shoulda and Mocha. RR is really interesting as well but I haven’t fully switched to it. I suspect I might replace Mocha with it at some point in the near future.

I typically start by writing model tests. They’re called unit tests in Rails, but they’re rarely isolated unit tests. Ideally, as much logic as possible lives in the models, so I spend a lot of time here. By far more than anywhere else. I do everything test-driven unless I’m hacking something together quickly. Even then, I usually reach a point where I wish I was doing things test-driven and switch.

Ideally, your controllers are going to be tiny. They’re also likely to be composed of calls to objects and methods from your model layer. So, though I do functional (controller) tests, I try to minimize the need for them. If you’re already testing a method in a model, you don’t need to duplicate that test in a controller if the controller is simply calling the model. You do need to make sure the controller is doing its job well. That’s where things like mocks come in. I use mocks not so much to avoid calling code in the dependencies an object has (such as the classic credit card processor example) but more to allow me to specify a process in terms of what the code I’m testing is supposed to do vs. how it does it.

Ultimately, it’s a good idea for someone to actually write real automated tests. These aren’t the ones I do unless I’m playing the role of tester. For example, Selenium is an excellent tool for really testing a Rails app. It’s great to have automated tests at that level that run with a continuous build and so on. I don’t usually actually do that so it’s not part of my process per se, but I’d always advocate that it be part of a team’s process. I don’t get much value out of trying to do selenium-first development, though, for example.

What’s the most interesting thing you’ve discovered about testing recently?

I wouldn’t call it a discovery, but I’m starting to change a long held opinion about the value of tests. I used to think of them as executable requirements specifications. I know a lot of people do. I’m from the school of TDD that blossomed into what’s now called BDD. BDD people even tend to use the word “specify” when they talk about tests they write up front.

That always sounded like a great idea. Requirements docs you can execute for validity. In my experience it rarely works out that way, though. Tests don’t end up being readable like english. RSpec and especially Cucumber are a step in that direction, but I’m starting to believe that ultimately developer tests should read like good code. Maybe a little different from normal good code, but they are code after all. And they’re not for validating the production code you write. They’re for motivating the production code you write.

So maybe the goals of executable requirements documentation and motivational specs are at odds with each other. And if the tests do the job of driving you to make well designed code but don’t really read like requirements documentation afterward, that’s nothing to feel bad about.

Maybe it’s even OK to throw away the tests eventually in the same way we used to throw away the code in our main()-method scratch pads.

I’m pretty sure I’m exaggerating, but that’s the general idea.

Is there a tool you wish you had for testing, but which you don’t think currently exists?

Not really. Marcel Molina and I used to talk about how we longed for something that would sit in the middle of our objects and their method invocations, such that you could set up how you expect an object to behave and then verify, without overriding the method’s behaviors, that the object did the things you asked. Like a partial mock which doesn’t change the method implementations it’s setting expectations for.

RR now does this with its proxy implementation. Now that it exists, though I like the feature, it doesn’t feel like it filled a big hole. Go figure.

What advice would you give somebody looking to write more effective tests?

Assume that writing more effective tests means writing better code. Developer testing (which is what I do) isn’t primarily about verifying code. It’s about making great code. If you can’t test something, it might be your testing skills failing you but it’s probably your code code’s design. Testable code is almost always better code. Code written as a result of a series of failing tests is very likely (by definition) testable. (An example of a design choice in Ruby that might result from this sort of approach is to use more mixins and less inheritance.)

That being said, it’s easy to fall into a trap when you start coding this way. I often come across heavily TDD’d code with a huge focus on some trivial but easily tested feature. For example, you might start writing tests for the Ruby comparison method (<=>) or to_s on an object, simply because you know how to write tests for it. I’ve seen programmers spend a disproportionate amount of time testing and implementing ancillary features because they get the TDD (or BDD) bug and find a comfort zone to hang out in.

Instead, always focus on testing the core of your domain. Other tests are nice to have, but when you focus on the core of your domain (the features that define the product you’re implementing), you drive that domain model forward and avoid spinning your wheels on testing for the sake of testing. Kent Beck used to say “test everything that could possibly break”. That’s good advice, but I’d add “and actually matters”.

Before I get to this interview, a brief program note—if you are interested in participating in this interview series, or if you have a request for some Rails developer that you’d like me to pester with questions about testing, please let me know at railsprescriptions at gmail dot com.

On to the interivew. James Golick is the founder of GiraffeSoft, a boutique Rails consulting firm out of Montreal. He also maintains the James on Software blog. Within the Rails community, he’s the author of resource_controller, a common parent for RESTful controllers, and active_presenter, which allows you to create presenter objects out of aggregations of ActiveRecord models.

Most recently, James has created zebra, a test library for the quick creation of single line tests.

How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?

I worked at a few really cheaply run companies, where I was the one man technology department. I didn’t sleep much in those days.

In those days, I was always desperately looking for ways to get better at what I was doing, if only to get the occasional extra hour of sleep. At one point, I came accross an article about automated testing and TDD, and it all just made so much sense to me. When you’re that strapped for time, automation is a necessity. Once I was introduced to the idea that tests could be automated, too, I jumped all over it.

What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?

At giraffesoft, we’re very serious about TDD. So, naturally, everything we do is test-first.

Currently, we write extensive unit tests and functional tests. We’re slowly adding cucumber to the mix. I’m definitely sold on the benefits of acceptance test driven development. So, that’s where we’re going.

We use Shoulda, largely because of its macros. Duplication in tests becomes incredibly tedious. So, expressing certain kinds of repetitive tests as one-liners is a huge win.

However, as I mentioned in the release announcement for zebra, we’ve started to feel the burn with Shoulda’s macros in certain situations. Often, test failures result in completely useless backtraces and the string programming catches up to you after a while.

So, we’re currently moving towards a context & zebra stack to replace Shoulda.

What’s the most interesting thing you’ve discovered about testing recently?

Using Jay Fields’ expectations testing framework completely changed the way that I approach unit tests. Not having to describe each test in english is incredibly liberating.

If your code is readable, you shouldn’t need to document it at a micro level, except in special cases. Your code probably isn’t littered with comments describing every couple of lines. So, why should your tests be?

Is there a tool you wish you had for testing, but which you don’t think currently exists?

If you’d asked me 2 weeks ago, I’d have replied, something that would allow me to write more expectations-like tests in my every day hacking. That’s what zebra is. So, now I’m feeling pretty happy about my toolset and where things are headed.

What advice would you give somebody looking to write more effective tests?

Be pragmatic. I get a lot of questions about the “proper” way to mock or stub something, for example. Stop worrying about getting things “right” and try to make judgements that get you the best possible test, while striking a balance between productivity and the longevity of the test.

Additionally, I’d really encourage people to use expectations for a small project – a rails plugin or something. For various reasons, you probably won’t want to use it for every day stuff, but you might learn a lot from giving it a serious look.

Next up on the testing interviews is Ryan Bates. Ryan runs Railscasts, a weekly screencast on a new Rails topic that is simply one of the best ongoing sources for tutorials about Rails. Seriously, if you aren’t familiar with it, drop everything and prepare to spend some time watching his videos. Ryan has also done two screencast series for Pragmatic, Everyday Active Record and Mastering Rails Forms, both of which are available at pragmatic.tv.

On a related topic, I didn’t introduce myself to Ryan at last year’s RailsConf, even though I walked past him a few times. This is because every time I walked by him, people were coming up to him and thanking for all the great Railscasts. So, thanks Ryan.

How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?

About 6 years ago I read the book Extreme Programming by Kent Beck. This sparked my interest in testing (specifically TDD), but I could not find many practical examples of the practice. I spent some time researching the topic but honestly did not “get” it until Rails came along. Rails provided practical testing patterns which were fairly easy to follow.

I find the most difficult part of testing is coming up with a pattern which works well for a specific situation. Once that is done, adding similar tests becomes very easy, and that is where it starts to really pay off.

What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?

I primarily use RSpec for testing, however my tests are quite different than what they recommend. For one thing, I use controller tests like functional and integration tests. That is, they execute the entire stack (including models and views). I don’t test views exclusively beyond this because I find that requires too many mocks and leads to brittle tests.

I test models and helpers exclusively (like unit tests). That is, I test each method on its own to ensure it functions properly. My theory is, the more deeper a piece of code is, then the more it is used by various parts of the application, and therefore should have better test coverage.

Fixtures are kind of interesting. I don’t use them at all in unit tests, instead I prefer factories (factory_girl) for this. However I do use fixtures in controller tests as filler data to help catch errors. Each fixture usually does not have more than two records, and I often leave them with their default generated content.

What’s the most interesting thing you’ve discovered about testing recently?

I just recently discovered the RR mocking framework and I like its syntax more than Mocha. However I haven’t moved many projects over to it yet, so I can’t say how well it works in real world use.

Is there a tool you wish you had for testing, but which you don’t think currently exists?

Perhaps a tool for testing private methods and accessing instance variables in a clean way. I know in theory tests shouldn’t need to do this, but I would still find it useful. There may already be one out there, I haven’t looked much.

Beyond this I would love to see more documentation, examples and experiments done with regard to testing. I still feel it is a fairly unfamiliar territory in the programming world, and everyone seems to have their own way of doing things.

What advice would you give somebody looking to write more effective tests?

Be careful with mocks and stubs. They are often an easy, immediate solution but can lead to brittle or deceiving tests. Only use them if you can find no other cleaner way to test something.

Overall, don’t give up on testing if you don’t grasp it right away. Try a technique for a while, if it doesn’t work with your flow, try something else. Don’t feel bad if you find testing is hard – it really is. But it is so worth it.

It’s time for another episode of Ask A Tester Person…

This is from Jônatas Paganini, writing from Brazil—I’ve paraphrased his question slightly, I hope I’m preserving the original intent.

In response to my advice on Page 11 of Getting Started about not writing too much code without tests, Jônatas asks about the opposite process.

What do you think about write a lot of tests before implementing the code? When running those tests, some of them break and I need to start fixing those tests. Is this a bad practice? I think that when I write just one test at a time I lose focus more easily.

Great question. I have several overlapping responses to this that I hope I can make coherent.

First off, as much as I love talking about theory in software development, practicality is obviously more important. If something works for you, then it works, and you don’t necessarily need to justify it. (That said, it’s always valuable to try new things to find out if there’s another way that works even better…)

As for this specific practice of writing multiple tests before writing code, my basic response is that I’ve generally regretted it the times that I’ve tried it, but that there are cases where it’s appropriate. It’s a much better practice to write a lot of tests first than it is to write a lot of code first.

The problems that I have with writing a bunch of tests up front are kind of subtle, and generally only an issue if you are writing, say, five or test tests in a row. Writing a couple of tests consecutively is usually not a problem, especially if the two tests are related enough to be covered by the same code.

The first problem is that writing all the tests up front tends to take away from the discovery part of the test-first process. Often, especially when writing code that’s not perfectly specified, writing a batch of code will suggest the next test, because there’s a clear weakness in the code that the next test will expose. This process will often lead to cleaner code at the end, I think, because of the way it gets built up in small steps. Writing all the tests at once makes it harder (but not impossible) to have the design emerge in the same way. For that reason, writing multiple tests at once is probably best done when dealing with a well-understood and specified algorithm.

Another way of describing this is to say that the more tests you write, the more the tests need to assume about the structure of code not yet written. You want to be careful that this doesn’t limit the final code. I will often wind up a test-first process with a more elegant solution than I would have imagined at the beginning. I think that going in one test at a time makes that easier.

The bigger problem, at least for me, is that if I write five tests in a row, I’d expect them all to fail. I find it hard to focus on multiple test failures at a time—if I do a refactor that breaks tests, I will often temporarily comment out tests so that I can deal with the broken tests one by one. Again, there are circumstances where this doesn’t apply. Sometimes, you’ll be writing error case tests that you are pretty sure that the code will handle. In that case, there’s less of a problem with writing multiple tests at once.

I do think the RSpec functionality of allowing you to specify the names of future tests without implementing them is often valuable as a partial version of this practice. By doing this you remember any error or unusual cases that you need to test, and you don’t get constrained by having to implement the tests against code that doesn’t yet exist.

Next up in the Testing Practices interview series is Gregg Pollack. Gregg is one of the proprietors of Rails Envy, and is one of the co-hosts of their Rails Envy Podcast, which mentioned Rails Prescriptions today—thanks Gregg!

Gregg is also one of the founding members of the Rails Activists. He’s also done a lot of video production, including some excellent ruby and rails screencasts, and his series condensing various Ruby and Rails conferences. Put it all that way, and he seems kind of busy, actually…

Take it away, Gregg…

How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?

I have to admit, my first 3 Rails projects didn’t contain any tests. I got into testing when RSpec started on the scene. RSpec just made more sense and the documentation on the RSpec website was really useful. Once I figured out how to use RSpec with autotest and growl, testing became much more fun.

I also ended up doing a talk at my local users group about how I came to love testing. It’s a little dated, but still quite relevant:

http://www.railsenvy.com/2007/10/4/how-i-learned-to-love-testing-presentation

What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?

I used to write isolated tests at the model and controller level using RSpec, properly mocking and stubbing at the controller level. Often times integration tests would get ignored.

Currently I’m working on a project using Shoulda for Model tests and Integration tests with Webrat. Yes, we’re not doing controller or view tests. So far I’m really liking the combination, and it seems to cover things pretty well without having to deal with much mocking and stubbing of controllers/view tests which have very little logic anyways.

What’s the most interesting thing you’ve discovered about testing recently?

Webrat rocks for integration tests.

Is there a tool you wish you had for testing, but which you don’t think currently exists?

Hmmm.. James Golick recently blogged about one line tests using a library called Zebra. He also argues that test names are essentially comments, and well.. too many comments are a code smell.

So what’s the alternative?

Creating a DSL that allows us to write tests in a way that doesn’t need names/comments. Shoulda gets us pretty close with all of it’s one line helpers like:

should_belong_to :user
should_have_many :tags, :through => :taggings
should_require_unique_attributes :title
should_require_attributes
should_only_allow_numeric_values_for :user_id

None of these tests need comments to figure out what they’re testing! I’d love to figure out a way to do more of this, and the Zebra library James put out is certainly a step in the right direction.

What advice would you give somebody looking to write more effective tests?

Integration tests are probably more important then anything else, and using a library like Webrat makes them very easy to do.

If you’re working with a team of people and you want to ensure you’re building a solid test library then a Continuous Integration server is imperative. Set it to run all your tests every time something is checked in and if it fails to email everyone. Make a rule that whoever checks in code that causes tests to fail has to buy everyone a round of beer, or has to perform some humiliating task. Checked in code that fails should be fixed immediately.

This would be a great time to mention that if you have an questions to ask about Rails testing, put them in the comments or email them to railsprescriptions at gmail dot com and I’ll try to address them here.

First up: Felipe Coury, from the comments:

Would you mind on elaborating on the factory approach versus traditional approach of using fixtures, either on a new article or even replying to this comment with references you may have?

Sure—this will be covered in some detail in the book itself, but it’s also worth some attention here.

The goal of the factory approach is to work around two of the weaknesses of fixtures: brittleness and opacity. These problems generally show up when you try to use fixtures for something that requires a lot of data, like search or report functionality. The workflow pattern often is to create a bunch of fixtures, then write tests that validate that, say, searching for “Smith”, returns the correct two records.

The first problem comes when somebody needs to add a new record to the fixture set to expose a different case for a different test, but the new global set of fixtures breaks existing reporting tests. Eventually this becomes a real pain. Also, when you read the test that says that searching for Smith returns two records, you need to back up and check out the fixture file to verify that. In my experience, that leads to a lot of tests where you write the test, and just fill in the value that the program spits out, rather than determining the results before writing the test.

The alternate approach is to create separate data for each individual test. The goal of the various factory tools (Factory Girl, Fixture Replacement, and Machinist are three to look at) is to make creating data for each test easy enough to make it a viable option for complicated testing.

All three of these tools are similar enough for my purposes here—essentially you have a file where you define default templates for each ActiveRecord model in your system plus a factory method that creates a new instance based on the default values. The default values can be dynamic (Faker works nicely here to create random structured data), and you can override any value in the template with the actual value you need for the test.

The goal here is to get new objects with a minimum of typing and, more importantly, with the key values needed for the test highlighted and the values that are irrelevant to the test in the background. This can make the test much easier to read. The pattern is to create a minimum amount of data for each test, focused on exposing the specific issue under test. Here’s an example, using Machinist syntax.

  test "I have two doggies" do
    @doggie = Dog.make(:name => "Mr. Puddles")
    @other_doggie = Dog.make(:name => "Rex")
    assert_equal(2, Dog.all.size)
  end

The advantage is that you can create exactly the data you need for each test easily enough that it doesn’t seem like a burden. Each test is independent, and easy to read. If you do wind up needing the same setup in multiple tests, it’s easy enough to move the calls to a common method, or use one of the various context tools.

  context "with two doggies" do
    setup do
      @doggie = Dog.make(:name => "Mr. Puddles")
      @other_doggie = Dog.make(:name => "Rex")
    end

    test "the doggies can be friends" 
      @doggie.befriend(@other_doggie)
    end
  end

There are two downsides that I have encountered. One is speed—fixtures by default use database transactions to optimize loading, factories generally don’t. This can slow things down, but if you are using the factories to create only the minimum amount of data for each test, you should still be okay. Similarly, it’s easy for the factory setup data to get so complex that it winds up in it’s own method, then it’s own module, which can take you close to the hard-to-read nature of fixture tests. I still think the factory tests are better, because you can group data from different models in the same file, but it’s something to watch out for.

Hope that helps, let me know if there are other questions you want me to cover here.

First up on the Testing Practices Interview series is Geoffrey Grosenbach, Senior Visionary of Topfunky, and also the person behind PeepCode. Geoffrey blogs at http://nubyonrails.com/, and is responsible for the gruff graph generator and, most recently, a task tracker based on David Seah’s Online CEO. He was also kind enough to write his responses in Textile, which I heartily endorse. Take it away…

How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?

I remember watching tests run during installation back when I was using Perl (wouldn’t that be nice if RubyGems optionally ran their test suite during installation?). But it seemed like an advanced topic that only some programmers did.

When I started using Ruby and met the testing fanatics at Seattle.rb, I started to understand what it was about and why I might want to write a test.

Three things helped me get started with test-driven development:

• Watching other people do it.
• Writing a graphics library (gruff). I needed to generate a bunch of samples with various data inputs and Test::Unit was a great way to do it.
• Finally, I started out by writing a single test for an existing Rails app whenever I encountered a bug. It gave me peace of mind.

Since then, I’ve appreciated the process of thinking that I get into when I code test-first.

What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?

I’ve tried several libraries and tools.

I started with Test::Unit and still use it on some existing projects.

For a while I used Jay Fields’ Unit-Record style of separating out unit and functional tests for both models and controllers (see also). It also provided a nice test method that took a string and a block of assertions (similar to what’s in Rails now).

I’ve used RSpec’s ability to add should syntax into Test::Unit.

I’m currently happy with straightforward RSpec. I also have some Test::Unit integration tests in my Rails apps but have also used RSpec User Stories and their replacement, Cucumber. Given the fact that I’m both the designer and coder of most of my apps, I don’t get much benefit from Cucumber. But I can see how it would be useful to people working for a semi-technical client.

Honestly, it’s a bit overwhelming with all the options out there!

When I’m actually coding, I’ll use Eric Hodel’s ubiquitous autotest or rstakeout to run my test suite.

What’s the most interesting thing you’ve discovered about testing recently?

I’ve been working on a small command-line Objective-C app and am experimenting with using Ruby to run a suite against the command-line app to check the inputs and outputs. Ruby is useful that way and works better for me than trying to use Objective-C for the same purpose.

Is there a tool you wish you had for testing, but which you don’t think currently exists?

I can’t think of one. Usually I end up writing a tool if I need it and I can’t find it anywhere, such as test_benchmark to show individual test runtimes for Test::Unit. Someone recently imported it to GitHub as well.

What advice would you give somebody looking to write more effective tests?

Find a mentor. Work with someone else who is doing it. Concentrate on testing the effects of your code, not the way they are implemented. It’s easy to write tests that really don’t do anything and won’t reveal meaningful changes in the code if it stops behaving properly.

Some test-related Rails links for you to chew on:

Pat Maddox on testing controllers. Key quote:

Controller specs don’t matter, because controllers don’t matter

In other words, all your complicated code should be in the model or in a mini-framework like James Golick’s resource_controller. If that’s true, then your controller tests become similar to integration tests.

The post has some nice examples of using Cucumber for controller testing—what I’ve called skeleton controller tests in other contexts.

Speaking of James Golick, he has a new tool called Zebra, which integrates with Context or Shoulda to add easy to write one line assertion tests inside a context: expect { @post.to be_editable_by(@author) }.

So far, I haven’t been the biggest advocate of the one-assertion-per-test style, but this syntax is nice, and I think I’ll be trying this out as soon as I can.

Meantime Stephen Boisvert was kind enough to link to the Patfinder cross-post of my interview with myself, saying:

Strangely light on depth and details given the interviewer’s relationship to the interviewee

It’s funny because it’s true! More details coming on the blog and the testing book. Also, thanks for saying nice things about my book—always nice to hear.