Jamie Dobson writes this piece of "reality" with respect to statistics/metrics and numbers.
“... that human beings will always work toward their defined success criteria."
True and very revealing for all metric enthusiasts. Just let people know what they will be measured on - they will modify their work pattern and output to suite positively on the measurement criteria. For example, a test team is measured on number of bugs the team logs, you will see more and more bugs and if a test team is measured on number of test cases they execute, you will see testers executing increasing number of test cases.
One thing that happens is what I refer as "goal displacement" - Goal of doing “required” work getting displaced by doing *that* work in *that* way as described/interpreted by measurement criteria. Can you see a problem here?
When working with social/cultural setup involving human beings, introduction of "monitoring/measuring", typically causes a "shift" in a overall behavior of the group towards "what is being measured" instead of "what is required". We tend to believe that people behave the same with or without a measurement system in place – we are wrong
This is the side effect that I am referring when a metrics program is introduced in a software project setup....
Are you aware of this? What steps can be taken to address the side effects?
Shrini
Thursday, June 26, 2008
Tuesday, June 24, 2008
Software Testing certifications Part II
Dr Cem Kaner posted a note to Software testing yahoo group on the topic “software testing certifications”. I thought, it would really make lots of sense and value to a discussion about software testing certifications to share those views here.
Dr Kaner quotes following in his note.
http://www.channelinsider.com/c/a/Careers/VARs-IT-Certs-More-About-Marketing-Less-About-Skills/
Continuing my thoughts on certifications, here is something that I would like to add … on the basis of Dr Kaner’s notes.
1. Certifications have value as “marketing” aid and most confuse them to be as means of getting knowledge or experience or learning.
2. IT organizations and service providers use their “certified staff” as “proof” of their well trained staff to their clients.
3. Certifications do have place in hiring. Whether you like it or not, organizations still use certifications as main filtering mechanisms in hiring just like a college engineering degree.
4. Certifications matter for those who are in the initial stages of their career especially those who are looking to get their foot in testing field. Most of the time, certifications get them a call to the interview.
5. Certifications can get you an interview call, might even get you a job but there after it is your skill and work that “keeps” you on job. Do not mistake certification for life time warranty for the job.
6. One very common argument in favor of certification is that “certification help in knowing the testing vocabulary” – This is true to some extent. But while going for certification with this objective, keep in mind that - there are no universally accepted authorities that define and mandate testing terminologies and terms and practices vary across the board.
7. Certification enthusiasts claim that certifications are means of learning and gaining knowledge in the subject. WRONG … there are better ways of studying and learning than going for certification
8. With the help of internet, thanks for Google and other search engines, today the information is everywhere, just look around you can learn lot and gain knowledge by effectively searching the web, reading blogs, writing blog and engaging in conversation with other in the community.
Now about let me talk about those certifications that hold value in today’s world
• CISSP (Certified Information Systems Security Professional] To earn a CISSP, candidates must have five years of experience and endorsement from any professional certified by (ISC)2, the organization that awards CISSP certifications.
• CCNA (Cisco Certified Internetwork Expert]) – this one especially my favorite as it requires the candidate to demonstrate the knowledge as part of the exam in a lab environment – e.g fixing a faulty router.
To summarize, certifications tend to be of value for some (hiring managers and new entrants) and there are some examples of good certifications that test the skill of the candidate. Do not confuse certifications to “learning” and “knowledge” – most of the current software testing certifications are to be used as “marketing” tools.
Dr Kaner quotes following in his note.
http://www.channelinsider.com/c/a/Careers/VARs-IT-Certs-More-About-Marketing-Less-About-Skills/
Continuing my thoughts on certifications, here is something that I would like to add … on the basis of Dr Kaner’s notes.
1. Certifications have value as “marketing” aid and most confuse them to be as means of getting knowledge or experience or learning.
2. IT organizations and service providers use their “certified staff” as “proof” of their well trained staff to their clients.
3. Certifications do have place in hiring. Whether you like it or not, organizations still use certifications as main filtering mechanisms in hiring just like a college engineering degree.
4. Certifications matter for those who are in the initial stages of their career especially those who are looking to get their foot in testing field. Most of the time, certifications get them a call to the interview.
5. Certifications can get you an interview call, might even get you a job but there after it is your skill and work that “keeps” you on job. Do not mistake certification for life time warranty for the job.
6. One very common argument in favor of certification is that “certification help in knowing the testing vocabulary” – This is true to some extent. But while going for certification with this objective, keep in mind that - there are no universally accepted authorities that define and mandate testing terminologies and terms and practices vary across the board.
7. Certification enthusiasts claim that certifications are means of learning and gaining knowledge in the subject. WRONG … there are better ways of studying and learning than going for certification
8. With the help of internet, thanks for Google and other search engines, today the information is everywhere, just look around you can learn lot and gain knowledge by effectively searching the web, reading blogs, writing blog and engaging in conversation with other in the community.
Now about let me talk about those certifications that hold value in today’s world
• CISSP (Certified Information Systems Security Professional] To earn a CISSP, candidates must have five years of experience and endorsement from any professional certified by (ISC)2, the organization that awards CISSP certifications.
• CCNA (Cisco Certified Internetwork Expert]) – this one especially my favorite as it requires the candidate to demonstrate the knowledge as part of the exam in a lab environment – e.g fixing a faulty router.
To summarize, certifications tend to be of value for some (hiring managers and new entrants) and there are some examples of good certifications that test the skill of the candidate. Do not confuse certifications to “learning” and “knowledge” – most of the current software testing certifications are to be used as “marketing” tools.
Friday, June 13, 2008
What if Automation finds bugs ....? Good thing or bad thing?
Ryan in response to my post on "cycle time reduction and automation", mentions that "It is an accurate statement that automation will not improve cycle time if it finds bugs, where bugs would have otherwise gone undetected. However, I believe the claims automation vendors make, is based on the fact that manual testing is also going to uncover those bugs."
So, when automation finds bugs, your cycle time increases. When people claim cycle time reduction, they "overlook" this fact. why? Those bugs will be found by manual testing too. This may or may not be the case. Bugs discovered by human testing and Automation tend to be of different types.
Now, let us track that trail of what happens a bug is discovered -
In automation - situation could be bit tricky especially when the automation tests, logs are bigger. An error/bug reported by an automation bug needs to be checked to see if it is a bug in automation code or a bug in application or bug in data setup or some timing or synchronization related problem (in GUI automation scenario). Let us say you have 5-7 pages of log file - you will have to scan/read through the log file an locate the bug. You might have to do execute failed automated test manually (and corresponding data setup etc).
In manual testing, human tester can easily trace and follow the bug trail and document the bug. At a high level, bug investigation and isolation tasks tend to become relatively low.
Hence, when automation discovers a bug - things get really problematic.
If one were to cut down cycle time by automation or otherwise, they HAVE to make sure either "no bugs are discovered" or "any discovered bugs are IGNORED" or "bugs that are discovered, if fixed, not tested again and other regression testing is done ....
Can automation control or influence any of above events - prevents bugs being discovered or igonore the bugs if accidently discovered or mandate that bugs fixes will not be subsequently tested?
For the sake of argument, let us suppose that both human test cycle and Automation find same number of bugs ... and take out "bugs" portion of test cycle, how automation can save test cycle time? On what parameters this cycle time reduction by Automation depends ?
Type of test - nature of interactions between "test execution agent" (human or an automated script) and nature of verifications (during and post execution).
Shrini
So, when automation finds bugs, your cycle time increases. When people claim cycle time reduction, they "overlook" this fact. why? Those bugs will be found by manual testing too. This may or may not be the case. Bugs discovered by human testing and Automation tend to be of different types.
Now, let us track that trail of what happens a bug is discovered -
In automation - situation could be bit tricky especially when the automation tests, logs are bigger. An error/bug reported by an automation bug needs to be checked to see if it is a bug in automation code or a bug in application or bug in data setup or some timing or synchronization related problem (in GUI automation scenario). Let us say you have 5-7 pages of log file - you will have to scan/read through the log file an locate the bug. You might have to do execute failed automated test manually (and corresponding data setup etc).
In manual testing, human tester can easily trace and follow the bug trail and document the bug. At a high level, bug investigation and isolation tasks tend to become relatively low.
Hence, when automation discovers a bug - things get really problematic.
If one were to cut down cycle time by automation or otherwise, they HAVE to make sure either "no bugs are discovered" or "any discovered bugs are IGNORED" or "bugs that are discovered, if fixed, not tested again and other regression testing is done ....
Can automation control or influence any of above events - prevents bugs being discovered or igonore the bugs if accidently discovered or mandate that bugs fixes will not be subsequently tested?
For the sake of argument, let us suppose that both human test cycle and Automation find same number of bugs ... and take out "bugs" portion of test cycle, how automation can save test cycle time? On what parameters this cycle time reduction by Automation depends ?
Type of test - nature of interactions between "test execution agent" (human or an automated script) and nature of verifications (during and post execution).
- GUI Forms with lots of data input fields - can result in quick form fill tests when automated (zero think time).
- Tests that require longer processing time can not gain from automation as automation can not speed up the processing time.
- Tests that require visual inspection - window titles, error messages, color, tool tip and other GUI centric elements - are better tested manunally as programmatic tests would mean lots of investment. Human testes are quicker and cheaper in such cases.
- Result verification that requires detailed analysis, text file processing, database check etc are good candidates for gaining cycle time.
Shrini
A catalogue of Test Automation Benefits ...
Other day, some one asked me “in your opinion, what are the real benefits of automation”. That triggered ideas for me to write this post… Let me attempt to consolidate and list all the benefits people claim around “automation”
Real: In my opinion, these are “real” and achievable benefits.
- Consistent and Accurate Test results (nearly free from human errors) – when accuracy and accurate results are important? – Numerical calculations
- Untiring and can be run for long hours without any loss of efficiency
- Quick – No think time while running tests. (Computer program does not think – So not useful in those cases where you need to think as you execute. How does automation program help you to gain speed when thinking is required?)
- Supplements human ability to spot software bugs
- Helps to run big number of data combinations – can test robustness
- Helps in multi platform combinations (OS/browser/database and other application setups)
- Repeat the testing (test execution) done for configuration A in Configuration B also.
- Hence increased Test coverage
- Following non test execution tasks
- Generate and manage special set of test data
- Large Volume Test comparisons
- Automated workflows and alerts
- Environment setup
Fake: These benefits are on transition point – where the focus slowly starts drifting from “real” to “imaginary” and hence “fake”. These are false promises that are realistically not possible.
- Improvement in application quality (automation can not improve application quality – even testing can not … only developers and business analysts can)
- Improvement in Test process (Test process is pre-requisite for automation)
- Non technical people can do automation – no programming language required
Improved test planning (not sure how)
- By executing a set of test cases for a specified number of times – ROI from automation can be realized. Say in 10 executions –automation test pays for itself after that is “cost saving” all along
Conditional: These are the benefits that are realizable and could be reasonable but under a very strict set of conditions – “once in a while” cases. When these benefits are stated, mostly the conditions that must be fulfilled to realize these benefits, are not stated. This makes these benefits look as though there are real and universally applicable
- Simply – saves time.
- Test effort reduction
- Improved Time to market
- Improved test productivity
- 24x7 testing possible
- Knowledge retention
Totally outrageous: There are some really outrageous and are more or less like a typical sales pitch. People, who make and believe such claims, do not seem to understand human side of testing, testing itself and automation. They are just believe that machines are better than human testers.
- An automation test execution is equal to an hour of skilled human testing
- Automation can replicate human interactions
- Reduced dependency on human testers
- Solves problem of resource crunch and less time available for testing
- Reduced defect rate – fewer defects
Anything I missed?
Real: In my opinion, these are “real” and achievable benefits.
- Consistent and Accurate Test results (nearly free from human errors) – when accuracy and accurate results are important? – Numerical calculations
- Untiring and can be run for long hours without any loss of efficiency
- Quick – No think time while running tests. (Computer program does not think – So not useful in those cases where you need to think as you execute. How does automation program help you to gain speed when thinking is required?)
- Supplements human ability to spot software bugs
- Helps to run big number of data combinations – can test robustness
- Helps in multi platform combinations (OS/browser/database and other application setups)
- Repeat the testing (test execution) done for configuration A in Configuration B also.
- Hence increased Test coverage
- Following non test execution tasks
- Generate and manage special set of test data
- Large Volume Test comparisons
- Automated workflows and alerts
- Environment setup
Fake: These benefits are on transition point – where the focus slowly starts drifting from “real” to “imaginary” and hence “fake”. These are false promises that are realistically not possible.
- Improvement in application quality (automation can not improve application quality – even testing can not … only developers and business analysts can)
- Improvement in Test process (Test process is pre-requisite for automation)
- Non technical people can do automation – no programming language required
Improved test planning (not sure how)
- By executing a set of test cases for a specified number of times – ROI from automation can be realized. Say in 10 executions –automation test pays for itself after that is “cost saving” all along
Conditional: These are the benefits that are realizable and could be reasonable but under a very strict set of conditions – “once in a while” cases. When these benefits are stated, mostly the conditions that must be fulfilled to realize these benefits, are not stated. This makes these benefits look as though there are real and universally applicable
- Simply – saves time.
- Test effort reduction
- Improved Time to market
- Improved test productivity
- 24x7 testing possible
- Knowledge retention
Totally outrageous: There are some really outrageous and are more or less like a typical sales pitch. People, who make and believe such claims, do not seem to understand human side of testing, testing itself and automation. They are just believe that machines are better than human testers.
- An automation test execution is equal to an hour of skilled human testing
- Automation can replicate human interactions
- Reduced dependency on human testers
- Solves problem of resource crunch and less time available for testing
- Reduced defect rate – fewer defects
Anything I missed?
Friday, June 06, 2008
What is a bug ... A new meaning ...
Let me give a try to this one-liner ...(short post)
"A software bug is a reflection of the mind of a confused human user"
Analyse this statement .....
[Some updates]
Above one-liner proposed by me, seem to have generated interest in some ... Let me clarify further ....
This one liner of mine is a beginning of an effort to understand human thinking process while he/she sees a bug. A human goes through a series of emotions while dealing with a software bug. A dominant emotion among these is "confusion" - state of perplexity, chaos, uncertainty.
Let us say you are running a test, observing what is happening and doing a quick comparison with what were expecting ... Suddenly something "unexpected" happens, your mind starts remodelling all that is happening, you were expecting "x" to happen, where as you are seeing "a", "b" and "c" are happening - that is contradicting your model .... your heart beat goes up, blood pressure goes up ... lots of physiological changes happen ... your brain tries to stabilize, create a new model on the fly, comprehend a things on the ground and after a while, you calm down and begin to put up an explanation of what is happening and why? A bug report comes out at the end ....
So Bhargavi --- the life cycle of a bug begins with a "confused" mind of a tester. A confusion is a mental state where you are not able to comprehend and reason to the situations and problems that you are subjected to (imagine a traffic cop at traffic jam on road, a kid on first day in a new school etc). It is the confusion that triggers the thinking in the mind of tester, tester follows the trail of the thinking and builds up a new model and finally when is mind is settled, comes up with a explanation. So when a bugs gets nailed down, mind becomes calm.
Even an obvious bug - when you look at it deeply is a result of thinking process triggered by "confusion" what is expected and what is observed. As a normal human reaction to a state of confusion, you reason out the things and make them clear - then bug becomes obvious ...
If an obvious bug is at level "0", confused state is at "-1" level. Also the "sense" of confidence is always followed by a subtle at times quick state of confusion. When try to open your car door in wrong way, open your cupboard keys in wrong position, when you are trying to open a door by pulling that opens only on "push" mode ... our mind goes through a quick state of confusion ... you quickly notice the situation and gain your calm.
It is deep cognitive process .... Psychologists will be able to explain better ... "Psychology of a BUG" ?
Shrini
"A software bug is a reflection of the mind of a confused human user"
Analyse this statement .....
[Some updates]
Above one-liner proposed by me, seem to have generated interest in some ... Let me clarify further ....
This one liner of mine is a beginning of an effort to understand human thinking process while he/she sees a bug. A human goes through a series of emotions while dealing with a software bug. A dominant emotion among these is "confusion" - state of perplexity, chaos, uncertainty.
Let us say you are running a test, observing what is happening and doing a quick comparison with what were expecting ... Suddenly something "unexpected" happens, your mind starts remodelling all that is happening, you were expecting "x" to happen, where as you are seeing "a", "b" and "c" are happening - that is contradicting your model .... your heart beat goes up, blood pressure goes up ... lots of physiological changes happen ... your brain tries to stabilize, create a new model on the fly, comprehend a things on the ground and after a while, you calm down and begin to put up an explanation of what is happening and why? A bug report comes out at the end ....
So Bhargavi --- the life cycle of a bug begins with a "confused" mind of a tester. A confusion is a mental state where you are not able to comprehend and reason to the situations and problems that you are subjected to (imagine a traffic cop at traffic jam on road, a kid on first day in a new school etc). It is the confusion that triggers the thinking in the mind of tester, tester follows the trail of the thinking and builds up a new model and finally when is mind is settled, comes up with a explanation. So when a bugs gets nailed down, mind becomes calm.
Even an obvious bug - when you look at it deeply is a result of thinking process triggered by "confusion" what is expected and what is observed. As a normal human reaction to a state of confusion, you reason out the things and make them clear - then bug becomes obvious ...
If an obvious bug is at level "0", confused state is at "-1" level. Also the "sense" of confidence is always followed by a subtle at times quick state of confusion. When try to open your car door in wrong way, open your cupboard keys in wrong position, when you are trying to open a door by pulling that opens only on "push" mode ... our mind goes through a quick state of confusion ... you quickly notice the situation and gain your calm.
It is deep cognitive process .... Psychologists will be able to explain better ... "Psychology of a BUG" ?
Shrini
Monday, June 02, 2008
Goal of Testing and a quotable quote
Michael Bolton in response to this post from Steve Rowe mentioned this gem …
“Do our automated tests take into account the notion that different people might value different things, and that one of the tester's primary goals is to recognize different constituencies and the ways in which their values might be threatened? “
Many people in our community think that as testers our goal is:
This is statement from Michael is an extension or probably logical conclusion of the famous statement that is often associated with testing “Be customer’s advocate” (or “think like customer”). Michael seems to suggest that it is not only customer whom we should consider, as testers we think about all stakeholders and explore what each of these stake holders value.
A stakeholder is a person is who is affected by the success or failure of a project or the Actions and inactions of a product or the effects of service – Cem Kaner
In one of the comments for the same post, I found another worth quoting statement from Ben Walther …
“There will always be inputs into your system that violate the assumptions made in building it. A computer generated test will not be able to violate such assumptions.”
Very true … that is why testing is so challenging and exciting
Shrini
“Do our automated tests take into account the notion that different people might value different things, and that one of the tester's primary goals is to recognize different constituencies and the ways in which their values might be threatened? “
Many people in our community think that as testers our goal is:
- To find bugs
- To prove that the application "works" as per the specifications.
- Run a bunch of tests and report the results,
- Develop some automation to speed up the execution of tests
This is statement from Michael is an extension or probably logical conclusion of the famous statement that is often associated with testing “Be customer’s advocate” (or “think like customer”). Michael seems to suggest that it is not only customer whom we should consider, as testers we think about all stakeholders and explore what each of these stake holders value.
A stakeholder is a person is who is affected by the success or failure of a project or the Actions and inactions of a product or the effects of service – Cem Kaner
In one of the comments for the same post, I found another worth quoting statement from Ben Walther …
“There will always be inputs into your system that violate the assumptions made in building it. A computer generated test will not be able to violate such assumptions.”
Very true … that is why testing is so challenging and exciting
Shrini
Subscribe to:
Posts (Atom)