There’s one important aspect of XP (and I think any agile software development process, if not any intelligent process) is to automate all tests. No ifs or buts about it. “But it can’t be done!” you say. Bullcrap. Anything can be automated if you try hard enough, and there are hundreds, if not thousands of tools out there to choose from.
“But, my customers won’t let me do that!”
Who are your customers to say you can’t write automated tests? What they SHOULD be concearned with is that you are delivering them a quality product in a timely matter. Quality being the keyword here… if you waste your development time manually testing the application (which can take quite a bit of time) you are already likely to miss a lot of different scenarios, and each time you need to retest, you’re going to spend the same amount of time manually testing it again… and again, you will probably miss something because let’s be honest, wouldn’t you rather be coding than manually testing stuff?
But then I’ll have to spend x amount of time writing tests, causing something that can be done in a day to be done in 3 days!
But if that extra time spent writing automated tests lets you test all of the scenarios in your application with the click of a button, is it really time wasted? If manually testing something takes 2 hours of your time, and you automate it so that you just click “test” and it does the same thing in even 5 minutes, and now you can repeat that test to your heart’s desire, isn’t that time well spent? If you can sit there and rerun your tests 10 times in 30 minutes, that’s possibly 20 hours of manual testing (which would never happen, so you won’t get the coverage).
The perception that it can take a lot more time to automate tests is concearning… because by automating tests we improve the quality of our product, and that improves our bottom line. So quit holding yourself down… quit thinking it can’t be done… and quit thinking that your customers won’t let you do it. Because all you need to do is tell your customer that you are doing this to ensure that they get top quality, and they’ll buy in.
Anyhow, some linkage to end this quick post. 
Often, it is not possible to test “all scenarios”, because there are not a finite number of things to test. When this is the case, often the best strategy is to focus on and automate the most “likely” scenarios. The best way to find the “likely” scenarios is often by hiring a testing expert (like James Bach, Elizabeth Hendrickson, Michael Bolton, etc.) to do some rapid/exploratory testing.
I agree with your sentiment (that automating tests is a valuable practice much, much more often than people think), but can’t agree with the absolute way you state it. Whether you should automate a given test is a function of how likely the test is to expose a bug, how severe the bug is, and the cost of automation. Even if automation is cheap (and if it isn’t, you should be working on making it so), some tests are not valuable enough to automate, but are valuable enough to execute quickly in order to get information once (of course, if you find yourself testing something more than once, it should be automated). The goal of testing is generally not absolute perfection, but getting ‘enough’ quality from the smallest possible investment (where the definition of ‘enough’ depends on your domain and business context).
I’d recommend subscribing to the blogs of the people I’ve mentioned above. I find us developers sometimes think of testing in a very limited scope. These bloggers have helped me expand my understanding of testing beyond the developer’s scope.
Cheers,
Ryan
P.S. Don’t forget about usability testing… can’t automate that either.
Another “pie in the sky” post… great. Have you ever tried to automate the testing of a Java ME application? Or one that is physically connected to a piece of electronics?
Please, do some research on some real projects, it will offer you a more nuanced vision on the world we live in. Not everybody is working on opensource headless applications, and you’re making the XP community look bad to a lot of people in search of better testing techniques.
@Ryan
I actually agree… there can be some things that are very difficult to automate the testing. Perhaps My point came across to zealous, which is what I was aiming for.
Sure, we probably can’t automate all tests, but I think taking the attitude of “Automate ALL Tests” will drive us to automate those that can. For example, at work we had something that everyone thought wasn’t testable, it was impossible they said. A co-worker and I decided to sit down during a personal development session and just tried it.
We were able to automate the said test, the test that was impossible to automate, the test there was no way it could be done, in 2 sessions.
@Mr.X
As I stated in my last comment, and as Ron points out in the article I linked to, yes, there are some things that you can’t write automated tests… but that doesn’t mean you shouldn’t try.
By keeping an “Automate All Tests” mindset, imho, helps you break past the “it can’t be done” barriers and quite possibly automate most of your tests, freeing time up for more productive tasks than manual testing.
Thanks,
James
My general take on test automation:
1. I have never (NEVER) regretted automating a test.
2. I have often regretted NOT’automating something sooner.
So, clearly I need to continue pushing further down the cost/benefit curve until #1 is no longer true. For now, based on the data above, if my initial answer is maybe, then I just go for it and assume it will be worthwhile.
What about answering the question “When not to automate ?”
Frederic Torres
http://www.InCisif.net
Web Testing with C# or VB.NET
@Mr.X
I developed a unit test framework for PPC (copy of JUnit), and with that framework I tested GSM/CDMA code with unit tests. The code would power on the radio hardware, make data connections, send SMS; and all was automated. I could start the tests on several devices one day, and get the results the next day. So this was not a pie in the sky note.