Comments on: Automate All Tests http://blog.james-carr.org/2007/05/02/automate-all-tests/ Rants and Musings of an Agile Developer Sat, 22 Nov 2008 07:55:52 +0000 http://wordpress.org/?v=2.6.1 By: redsolo http://blog.james-carr.org/2007/05/02/automate-all-tests/#comment-29597 redsolo Tue, 21 Aug 2007 11:52:00 +0000 http://blog.james-carr.org/?p=73#comment-29597 @Mr.X I developed a unit test framework for PPC (copy of JUnit), and with that framework I tested GSM/CDMA code with unit tests. The code would power on the radio hardware, make data connections, send SMS; and all was automated. I could start the tests on several devices one day, and get the results the next day. So this was not a pie in the sky note. @Mr.X

I developed a unit test framework for PPC (copy of JUnit), and with that framework I tested GSM/CDMA code with unit tests. The code would power on the radio hardware, make data connections, send SMS; and all was automated. I could start the tests on several devices one day, and get the results the next day. So this was not a pie in the sky note.

]]> By: Frederic Torres http://blog.james-carr.org/2007/05/02/automate-all-tests/#comment-23571 Frederic Torres Thu, 21 Jun 2007 03:13:21 +0000 http://blog.james-carr.org/?p=73#comment-23571 What about answering the question "When not to automate ?" Frederic Torres www.InCisif.net Web Testing with C# or VB.NET What about answering the question “When not to automate ?”

Frederic Torres
http://www.InCisif.net
Web Testing with C# or VB.NET

]]> By: Alex Miller http://blog.james-carr.org/2007/05/02/automate-all-tests/#comment-13680 Alex Miller Wed, 02 May 2007 23:48:09 +0000 http://blog.james-carr.org/?p=73#comment-13680 My general take on test automation: 1. I have never (NEVER) regretted automating a test. 2. I have often regretted NOT'automating something sooner. So, clearly I need to continue pushing further down the cost/benefit curve until #1 is no longer true. For now, based on the data above, if my initial answer is maybe, then I just go for it and assume it will be worthwhile. My general take on test automation:
1. I have never (NEVER) regretted automating a test.
2. I have often regretted NOT’automating something sooner.

So, clearly I need to continue pushing further down the cost/benefit curve until #1 is no longer true. For now, based on the data above, if my initial answer is maybe, then I just go for it and assume it will be worthwhile.

]]> By: James Carr http://blog.james-carr.org/2007/05/02/automate-all-tests/#comment-13671 James Carr Wed, 02 May 2007 19:51:49 +0000 http://blog.james-carr.org/?p=73#comment-13671 @Mr.X As I stated in my last comment, and as Ron points out in the article I linked to, yes, there are some things that you can't write automated tests... but that doesn't mean you shouldn't try. By keeping an "Automate All Tests" mindset, imho, helps you break past the "it can't be done" barriers and quite possibly automate most of your tests, freeing time up for more productive tasks than manual testing. Thanks, James @Mr.X

As I stated in my last comment, and as Ron points out in the article I linked to, yes, there are some things that you can’t write automated tests… but that doesn’t mean you shouldn’t try.

By keeping an “Automate All Tests” mindset, imho, helps you break past the “it can’t be done” barriers and quite possibly automate most of your tests, freeing time up for more productive tasks than manual testing.

Thanks,
James

]]> By: James Carr http://blog.james-carr.org/2007/05/02/automate-all-tests/#comment-13670 James Carr Wed, 02 May 2007 19:49:07 +0000 http://blog.james-carr.org/?p=73#comment-13670 @Ryan I actually agree... there can be some things that are very difficult to automate the testing. Perhaps My point came across to zealous, which is what I was aiming for. ;) Sure, we probably can't automate all tests, but I think taking the attitude of "Automate ALL Tests" will drive us to automate those that can. For example, at work we had something that everyone thought wasn't testable, it was impossible they said. A co-worker and I decided to sit down during a personal development session and just tried it. We were able to automate the said test, the test that was impossible to automate, the test there was no way it could be done, in 2 sessions. @Ryan

I actually agree… there can be some things that are very difficult to automate the testing. Perhaps My point came across to zealous, which is what I was aiming for. ;)

Sure, we probably can’t automate all tests, but I think taking the attitude of “Automate ALL Tests” will drive us to automate those that can. For example, at work we had something that everyone thought wasn’t testable, it was impossible they said. A co-worker and I decided to sit down during a personal development session and just tried it.

We were able to automate the said test, the test that was impossible to automate, the test there was no way it could be done, in 2 sessions.

]]> By: x http://blog.james-carr.org/2007/05/02/automate-all-tests/#comment-13660 x Wed, 02 May 2007 16:13:55 +0000 http://blog.james-carr.org/?p=73#comment-13660 Another "pie in the sky" post... great. Have you ever tried to automate the testing of a Java ME application? Or one that is physically connected to a piece of electronics? Please, do some research on some real projects, it will offer you a more nuanced vision on the world we live in. Not everybody is working on opensource headless applications, and you're making the XP community look bad to a lot of people in search of better testing techniques. Another “pie in the sky” post… great. Have you ever tried to automate the testing of a Java ME application? Or one that is physically connected to a piece of electronics?

Please, do some research on some real projects, it will offer you a more nuanced vision on the world we live in. Not everybody is working on opensource headless applications, and you’re making the XP community look bad to a lot of people in search of better testing techniques.

]]> By: Ryan Cooper http://blog.james-carr.org/2007/05/02/automate-all-tests/#comment-13658 Ryan Cooper Wed, 02 May 2007 15:57:17 +0000 http://blog.james-carr.org/?p=73#comment-13658 Often, it is not possible to test "all scenarios", because there are not a finite number of things to test. When this is the case, often the best strategy is to focus on and automate the most "likely" scenarios. The best way to find the "likely" scenarios is often by hiring a testing expert (like James Bach, Elizabeth Hendrickson, Michael Bolton, etc.) to do some rapid/exploratory testing. I agree with your sentiment (that automating tests is a valuable practice much, much more often than people think), but can't agree with the absolute way you state it. Whether you should automate a given test is a function of how likely the test is to expose a bug, how severe the bug is, and the cost of automation. Even if automation is cheap (and if it isn't, you should be working on making it so), some tests are not valuable enough to automate, but are valuable enough to execute quickly in order to get information once (of course, if you find yourself testing something more than once, it should be automated). The goal of testing is generally not absolute perfection, but getting 'enough' quality from the smallest possible investment (where the definition of 'enough' depends on your domain and business context). I'd recommend subscribing to the blogs of the people I've mentioned above. I find us developers sometimes think of testing in a very limited scope. These bloggers have helped me expand my understanding of testing beyond the developer's scope. Cheers, Ryan P.S. Don't forget about usability testing... can't automate that either. :) Often, it is not possible to test “all scenarios”, because there are not a finite number of things to test. When this is the case, often the best strategy is to focus on and automate the most “likely” scenarios. The best way to find the “likely” scenarios is often by hiring a testing expert (like James Bach, Elizabeth Hendrickson, Michael Bolton, etc.) to do some rapid/exploratory testing.

I agree with your sentiment (that automating tests is a valuable practice much, much more often than people think), but can’t agree with the absolute way you state it. Whether you should automate a given test is a function of how likely the test is to expose a bug, how severe the bug is, and the cost of automation. Even if automation is cheap (and if it isn’t, you should be working on making it so), some tests are not valuable enough to automate, but are valuable enough to execute quickly in order to get information once (of course, if you find yourself testing something more than once, it should be automated). The goal of testing is generally not absolute perfection, but getting ‘enough’ quality from the smallest possible investment (where the definition of ‘enough’ depends on your domain and business context).

I’d recommend subscribing to the blogs of the people I’ve mentioned above. I find us developers sometimes think of testing in a very limited scope. These bloggers have helped me expand my understanding of testing beyond the developer’s scope.

Cheers,
Ryan

P.S. Don’t forget about usability testing… can’t automate that either. :)

]]>