When Not to Automate
Edit: Don’t get me wrong… I still stand by my motto to “Automate All Tests” … my viewpoint here is that sometimes you should use common sense if an automated test provides zero value. 
In a previous post, I ranted that we should work to “Automate All Tests.” And I still stand by my original sentiment… as professionals, we should always strive to write as many automated tests as we can to ensure quality. One posted a comment asking “How about when not to automate?” Well, I think this iteration I finally came across the scenario in a project when you shouldn’t automate.
Part of this iteration we had a few cards related to changing the look and feel of a few pages on our site (which all contained forms) and updating some of the static text. We made our changes, uploaded the images, and tested across browsers and made sure everything still worked. Then our QA (who is also a little new) asked, “Shouldn’t we have some automated tests for this card?” I shook my head no. “But shouldn’t we try to automate tests for everything?”
I think this is the case where you shouldn’t… and I’ll explain why. For starters, the UI is always a volatile place that can change frequently.. writing tests around that ensures you’ll have broken tests in the near future. Second, what business logic are you testing? Like I said before, writing a test for static text is silly… there is zero effort in adding that text, writing the test would be more time intensive, and again your test will break next time the customer wants to change it (and FFS, TDD’ing text is really plain out silly). And finally, building off the last reason, the domain is what matters most… that’s why we spend much effort decoupling our view from the model so that we can test our model in isolation.
This is why I tend to really criticize Fitnesse fixtures like HtmlFixture or even Selenium to an extent (I have found a lot of GOOD use for that tool however)… the domain language of these tests tend to be very web developer centric (elements, ids, etc) and as a result largely focus on the UI so much that they completely ignore the domain logic happening behind the scenes. IMHO a test to make sure form fields are present is a waste of time. On the other hand, a test that fills out those form fields with different combinations of data, submits them, and verifies the expected outputs is a VERY worthwhile test.
Remember the key point is to ask yourself what benefit you gain from writing an automated test, and whether that benefit is worthwhile. For me, a simple gander at the UI and asking the customer to confirm it looks good is “good enough” and gives me time to focus on writing tests for and developing the domain layer more.
And the domain layer is the most important place in you application. Period.
If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!
September 14th, 2007 at 7:34 am
James,
I think it can be worthwhile to verify that a dynamically built form is build according to expectations, but I can see why you think it would be silly to verify that a statically-typed-in-by-the-programmer form would meet expectations. All you are doing is requiring them to type it twice - once for the test and once for the site/application, which would just turn into copy and paste.
It might be nice to automate testing that it is valid markup though. Even an existence test can be worthwhile for the times someone messes with it and removes a field that should be there (such as when you build a form, client has access to change it, and then removes a field that should always be there).
Regards,
Sam
September 14th, 2007 at 8:40 am
I kinda agree with the general point, but I feel that you’re missing one key benefit of automated tests: backup for refactoring. Even when a test seems dumb or unneeded, you never know when it can suddenly become the indicator that something went wrong in a refactoring session.
Simply put, the more tests I have in my project (even silly ones sometimes), the more confident I am when hacking and slashing in my code refactoring everything. I just run my full build, and if everything still passes, then I can commit to the source control and sleep well at night.
Granted, the above does not apply to “static text” testing, obviously. Such tests are really silly.
September 14th, 2007 at 8:48 am
James,
Your statement that “IMHO a test to make sure form fields are present is a waste of time” is without context.
If you are using a form builder class/module, then there is definitely a point to making sure that the expected fields are present in the form.
If it is a static, manually edited form, you might be right. However, I don’t see that it would be too much work to automate it. I would even argue that writing the test for it and running the test is much faster than manually verifying it. In addition to that, having such test in place could point you to the source of error pretty fast should that field be accidentally deleted, saving you some time hunting for bugs in the wrong places.
September 14th, 2007 at 8:56 am
Peter,
I agree… my point is I have come across UI tests that ONLY test that form fields are present. However, I have written tests that do fill out form fields, submit them, and check the output (dao, email, whatever) of the process and maybe the page the user ends up on. As I said:
I just take an issue with automated tests testing such things like images being present, static text, etc. Of course, this is at discretion… I say it’s dumb to test images, but what if it’s part of something like a map renderer? Like always, context matters.
Thanks,
James