A note on Test estimation ...

I was discussion with one of my colleague about test case estimation where a question was raised about “how to handle creep in terms of number of test cases? Let us say initially if we estimate x number of test cases and this number at later point of time becomes 2x”

My response was -

Estimation is always an iterative process. You typically make estimation in terms of test case early in the test cycle - that is in planning phase. Make all the stake holders clear that "Estimates are based on current understanding of the application and test requirements and are likely to change". Have this as main "Disclaimer" in the test plan or estimation document.

This will give flexibility later in the cycle to ask for more resources and time. If you make your PM and other stakeholders crib or complain - Tell them as test team progresses, test team will get more understanding and initial estimates are likely (mostly will) change. Just say "I told you so" and show them the line in the test plan.

This is diplomatic way of handling future uncertainties in test estimates

Other side of this situation is that - test cases number increasing from 2000 to 4000 will happen in test design phase. So you can write all those 4000 test cases if time permits and execute only important ones during execution phase.

So far, as I have seen in this industry - Test effort estimation happens by experience, manipulation and intelligent planning. It is more of negotiation and communication skills than any science or proven method.

When development - in spite having about half a dozen estimation techniques, international bodies or knowledge, certifications like PMP - fail more often in estimating development effort, we, in test while estimating, can start with some good value and keep it open for future updates.

Please spare testing community from getting subjected to scrutiny for estimation... We are learning

Shrini

Windows Registry hacks ..

Here is a lazy post but quite useful ....

Your one stop shop for all registry hacks on Windows

http://www.winguides.com/registry/

My Fav registry hack is to block access to specific hard drive (let us say C) to unauthorised users ....

Other one is preventing Right click on Desktop....

Shrini

automationjunkies ....

Found this interesting site --

http://www.automationjunkies.com/index.shtml

Looks bit outdated (appears to be updated last in 2004. But is a good collection of resources related to automation...

Check it out ...

Shrini

QA and Testing - Debate continues ...

Further to this discussion on QA vs Testing – Michael Bolton makes a very interesting statement about what QA can do which tester can not or not empowered to do.

For myself, I don't like the term "quality assurance", and will do what I can to make sure that I'm called a tester. Unless I have authority and control over schedule, budget, staffing, product content, and product direction, I don't have the ability to assure quality in a product. I can report on it, though--and that's what a tester does, in my view.

Here is the complete Google group discussion thread on the topic
http://groups.google.com/group/comp.software.testing/browse_thread/thread/71c357a1c4d1b342

Shrini

Estimation for Test Automation Part 1 ...

Test estimation itself is mystery or magic tool that every test and project manager is trying to master these days with very little success - I should say. Then Test estimation for Automation gets little more complicated as it involves other software - automation tool. It is like a estimating for full-fledged application development and testing project. Treat it like nothing less. Ask your manager or his/her colleague PM how they do estimation for complete project (dev and test) - Take some leaves out of their experience.

Some thoughts to get you started off :

1. Like software development project - automation also has its own similar lifecycle
a. Automation Planning
b. Test cases Analysis (= Requirements phase)
c. Automation Design (= Design Phase)
d. Coding and Unit Testing of Scripts
e. Automation builds
f. Source control and Testing, Defect management
g. Deployment on Test lab - try run - fix and re-run
h. Sign off
So be sure to factor in time for all these - just focusing on test case and their complexity is a surely leads to underestimation

2. Following are common and assumed to be factored by the PM - make sure you check whether your team is up there.
a. Required tools licenses
b. Team having training usage of QTP and some development experience
c. References like coding guidelines, Config Mgmt guidelines and other dos and donts kind of documents
d. Setting up of test environment - this is very imp. Mostly we assume that it is there and when you are starting the project you will spend more than you estimated time in getting Test environment up and running for whole group. Take this point little more serious if you are running an “offshore-onsite” type of automation.
e. Framework - supporting structure other than the tool - decide whether you need it or not
f. Other tools and software requirements like - VSS, Database, shared drive to keep common stuff etc

3. Test cases - you need to look at test case complexity from a different angle when you are automating. It does not help in classifying test cases as simple, complex etc. You should look it from over all development of common code perspective. Take a set of logically related test cases and think how many re-usable functions you will require - each for navigation, data input and verification. More the number of functions that a test case needs – more complex it will be for automate. Hence take more time. While estimating always consider a group of test cases not individuals. In a test case items that need to considered are - number of steps, number of inputs and number and type of verification points.

When you are asked for estimation for automation - ask for time to analyze the target test cases and then make judgment call. If you are asked to estimate in a quick and dirty way - shoot back asking - how much of error in estimation they are willing to take - 20-30%? Tell them you will refine your estimates once you have a complete look at test cases. As it happens in development you will revise your estimate after requirements (if it is allowed) - Do it after test case thorough analysis....

Rest in part 2 ...

Shrini

Pairwise Testing ...

This is an interesting topic in testing related to testing of a feature that is influenced bt multiple variables. Here is one blog post from Apoorva Joshi - which is like single reference that points out to several others notable one - each two Famous Michael's in testing community - Michael Hunter of microsoft and Michael Bolton.

http://criticsden.blogspot.com/2005/02/pairwise-testing.html

Right now, I am too anxious to make this post so that I can comeback later with my comments ...

Here are my quick questions about Pairwise testing
1. Why pair is imporatant? What about triplets or 4 variables at a time?
2. How is concept of "Orthogonal" or "Taguchi Arrays" related to Pairwise testing
I will study and come back on this ... Meanwhile enjoy reading above thread ...

Shrini

Server Virtualization

In computing, virtualization is the process of presenting a logical grouping or subset of computing resources so that they can be accessed in ways that give benefits over the original configuration. This new virtual view of the resources is not restricted by the implementation, geographic location or the physical configuration of underlying resources. Commonly virtualized resources include computing power and data storage.

A good example of virtualization is modern symmetric multiprocessing computer architectures that contain more than one CPU. Operating systems are usually configured in such a way that the multiple CPUs can be presented as a single processing unit. Thus software applications can be written for a single logical (virtual) processing unit, which is much simpler than having to work with a large number of different processor configurations.

Virtualization is about running an Operating System (the guest OS) on the top of another OS (the host OS). This technique enables running several virtual machines with different OSes at the same time on the same hardware. VMWare, MacOnLinux, and Xen are examples of virtualizer software. Virtualization requires guest OSes to be built for the host machine processor. It should not be confused with emulation, that do not have this requirement: When an OS runs on the top of a virtualizer, its code runs unchanged on the processor, whereas an emulator has to interpret to the guest OS code. MAME or Basilisk are examples of emulators. Binary compatibility is another different feature: it is the ability of an OS to run applications from another OS. For instance BSD systems are able to run Linux binaries.

Avenues for Virtualization software

1. MS Virtual Server 2005 - http://www.microsoft.com/windowsserversystem/virtualserver/default.mspx

2. VMWare – www.vmware.com
3. XenSource - http://www.xensource.com/ (from Xen Open Source community)


References:
http://www.virtualization.info/
A great article with an introduction to Virtualization [kernelthread.com]
Microsoft Virtual server Road Map :
http://www.entmag.com/reports/article.asp?EditorialsID=87


Advantages of Virtualization

1. Increase utilization of existing Server hardware
2. Easy maintenance
3. Help in Business continuity and Disaster Recovery initiatives


Shrini

Automation of Setup programs ..

Let me make a blanket statement - My first impression and view is that "Setup programs" in general are not suitable for automation. They are best tested manually - unless you work for company like InstallShield whose products themselves help in creating setup programs. Those seup programs having 5-6 steps and taking just folder name as input - are not going provide return on investment for automation.

You decide to play devil's advocate and say -"I don’t agree with you" and insist on automation - Here is one approach.

1. Identify how big is setup program - how many steps are there? 10? 20? or more than 20? what are the variations possible? 100+? If yes - automation will help you.

2. Now identify the most dense and cluttered screen/step in setup that takes large number of inputs. Automate that screen only and proceed and automate let us say top 5 critical screens with 5 scripts.

3. In my opinion - there is no need to automate the setup flow unless there are more than 20 steps and 100+ variations possible.

Look at the beauty and usefulness of such analysis - you are trying to do automation, in the process, you ask so many questions. Think and create those scenarios which otherwise would never have been explored if you are following a structured and scripted test plan. See the value here. At the end you may or may not automate all those scenarios but while trying to automate and tying to convince that automation is way to go - you have tested and enriched test scenarios.

To quote Michael Bolton (www.developsense.com), a friend and mentor - "Often automation, in itself may not lead to good testing or value directly but during the course of preparation whatever the analysis you do and questions you ask "How can I verify this"?”Why should I automate this?" "What can go wrong here" - are VALUABLE and should be done.

How? By always playing a devils advocate - "why" and "why not" If you stop questioning and just accept what is being told to you - you stop learning and cease to become a Tester...

Shrini

Shrini at STEP IN 

Tips for Developer Testing ....

Do stop by at this blog post to get Braidy tester Michael Hunter (Microsoft) tips for Developer Testing ...

http://blogs.msdn.com/micahel/archive/2006/01/25/TestingForDevelopers.aspx

Shrini

In quest of automation tools ...

As automation is catching up like wild fire in software testing field – people are frantically searching for cost efficient ways to do automation. Some cash rich companies are investing in industry standard and proven automation solutions from leading tools vendors like Mercury, Rational, Compuware and Segue – other “not_so_rich” companies are struggling around “open source” free tools. Some of the product companies like Microsoft, CISCO – invest in developing their own in house tools. So, broadly we have three categories of automation tools in testing – Commercial automation tools, Open source tools and in-house tools. The first two are available to testing community at large. The information about commercial tools is rather well known and is available at respective websites

Mercury - http://www.mercury.com

Compuware - http://www.compuware.com

Rational - http://www-306.ibm.com/software/awdtools/tester/functional/index.html
(It is quite surprising to see that the information about "once highly popular automation tool" Rational Robot - has gone so deeper into IBM site - it is hardly visible link on IBM main site)

Segue - http://www.segue.com


Here are some free tools on web (a partial list based on my own searching of such free tools).

1. Watir – Web application testing in Ruby http://wtr.rubyforge.org/
2. Watir Web Recorder - http://www.mjtnet.com/watir_webrecorder.htm
3. Open source testing tools http://opensourcetesting.org/
(Note that these are “TESTING” tools not “AUTOMATION” tools)
4. Selenium- A test tool for web applications- http://www.openqa.org/selenium/
5. TestMaker – Framework for Test automation of web based applications and Web services. http://www.pushtotest.com/Downloads/

This is ever growing list – Good thing is that more and more people are investing in developing Open source tools. This will build up pressure on Commercial automation tool vendor to offer tools that are cheaper price and rich in functionality.

Shrini

Further on the road in becoming a finest software tester ...

One of my blog reader asked my views on things that a good tester should invest on and personal traits and qualities of a good tester. I did write about it at my blog in this post. Here is a sequel to it

1. First important thing become a great testers is to question. Question around you anything - be it Door Knob, Watch, your vehicle, your access card, your rice cooker, Gas stove, TV, cell phone. Get curious about everything around you. Find bugs in everything that you see. World is a giant software and is full of bugs. Find bugs there. Then finding them in software will become fun.

2. Powerful observation: can you observe things that others don’t see? Can you notice that little color fading on the billboard hoarding? Can you find error or mistake in Prime ministers reported speech? How about in annual report of Infosys or IBM? Be aware of anything around see deep into it. Smart testers find bugs by observing carefully and are always curious about things - they never stop.

3. Memory and analytical skills. Take tests or training to improve memory. Solve puzzles, play chess, Suduko, jigsaw puzzles. Few of these tricks are mentioned in this blog post.

4. Becoming a good tester should be your goal - "logging tonnes of bugs" or "getting expertise in automation" - will come as side effects - they will be natural to you. Remember - there are no shortcuts - be lifetime student of learning, questioning and observation.

Have you read articles from James Bach and Michael Bolton on Critical thinking? If not, they should be good beginning. Don’t expect immediate results. Depending upon how committed are you and how fast you get into the mode of questioning and observation - it could take about 1-2 years to call yourself a descent tester. That is my rough estimate.

Current industry trends focus on process, factory thinking and want to make testing as routine and predictable - real testing is always interesting, instantaneous and dynamic. You can not do good testing by a following a “pre-laid out list” - You need to think.


Shrini

Testing ideas for Database Stored Procedures ...

I happen to see a query in Google group on software Testing regarding testing of stored procedures ( as a part of Database testing). Here are my views on this topic

In one way Stored Proc (SP) Testing is like a testing an API. So apply all the rules that you would apply for API - to SPs also. You can design test cases treating Stored Proc as black box - design all possible combinations of valid and invalid inputs and observer outputs. Be sensitive about the fact that when a SP is tested like an API – you will supply certain inputs which are otherwise not likely to be fed to it if used by a middleware component. So developer may reject a bug related to SP saying that SP will be never be called with such set of strange of inputs – client/middleware layer will filter all bad or unlikely inputs.

For structural Testing (white box) following things come to my mind...

1. You might want to consider measuring Cyclomatic complexity of the code (of loops in it). I believe there are some tools that measure this. Here are few links related to CC measurement. By the way this is also referred as "Code complexity". Higher the code complexity - higher the testing effort required to validate it.

http://www.sei.cmu.edu/str/descriptions/cyclomatic_body.html

http://www.linuxjournal.com/article/8035


2. Databases products like MS SQL server (oracle - not sure) provide monitoring / profiling tools to assist in the time taken for SP execution and other runtime parameters. This will give good idea about SP's Runtime performance. I used MS SQL server Profiler tool to monitor SP execution.

3. You could test SP for security/access control.


Shrini

QA!= Testing : Mother of all testing related debates is here ....

Whatever is QA, It is not testing – Cem Kaner
Do you want any better and authentic source of opinion on this topic? May be not. It is the time to open the eyes. Let us face, Testing has grown big enough to be separated from its Big brother QA. My sole aim of writing this post to educate the mass about the misconception caused by using wrong terminologies at all places.

My versions of differences:

QA: An entity that observes the activities of software engineering (while being performed or at the end of some logical milestone) and verifies that all documented or know rules and procedures are adhered. This entity does not “Touch” the work product being produced. It watches the process.

Test: Something that gets hands dirty, works with the product, uses it, abuses it, checks all relevant documents etc. Something that is similar to the development team that produced the work product – by from an evaluation angle.


These definitions are my versions and I have attempted in my best abilities to represent the mass opinion about these terms. I could be wrong … I am ready for a debate.


Problems of using QA word for testing:

Testing can not assure quality. “Testing” only measures or attempts to measure it. Quality has to be built from the beginning not to be tested later.
Assuring Quality is everybody’s (in the team) responsibility. How only testing team can be entrusted with that job? In true sense, it is management that is the real QA.
If it is strongly believed that Testing is responsible for ensuring the quality – then others can say – it is testing teams to job to ensure quality not ours.
Sets up wrong expectations about the role of testing.


Notable posts that discuss this topic

http://www.testingreflections.com/node/view/827
http://blogs.msdn.com/micahel/archive/2004/10/27/248564.aspx
http://www.stickyminds.com/sitewide.asp?ObjectId=3543&Function=DETAILBROWSE&ObjectType=COL

Shrini

AJAX - Making web experience more interactive ..

AJAX - A technology of bringing desktop software experience to web users. In this it is possible to do the things like editing data on client browser and get an update without web server resending the updated information. Be it “Editing photos on Yahoo photo albums” or using “Google maps” - it AJAX at work. Ajax is not a technology in itself, but a term that refers to the use of a group of technologies together.

AJAX - stands for "Asynchronous Java Script with XML". Java script is a default scripting model for web pages that is used to perform most of the client end data handling, rendering and other local processing. This when combined with XML can transfer the things greatly

AJAX is often considered as a threat to Macromedia's Flash Technology. Flash is more popular technology for rendering dynamic content in the form of multimedia, video based content at Web pages.

Companies like Microsoft, AOL and Amazon are already off the block in deploying this technology to spice up their Web sites. There is an increasing trend in web designers to embrace this. Ajax applications look almost as if they reside on the user's machine, rather than across the Internet on a server.

Read here more about it --
http://en.wikipedia.org/wiki/AJAX
http://www.adaptivepath.com/publications/essays/archives/000385.php

Happy Christmas and New Year 2006 ...

Shrini

Risk based Testing ....some points to ponder

Risk based Testing - I can see lots of eyes rolling and eye brows raising on seeing this term. Likes of Project managers and Delivery managers simply love to hate this term "Risk". My boss, Manjunath Hebbar, other day gave a this beautiful definition of risk "A risk is piece of information regarding tomorrow 's (future) problem as on today (present)" and he further said - in contrast this an issues is "problem in the hand". Well - this discussion with him was around "project management" not about testing.I am not that competent to talk about project management, so let me be happy with what I am good at - testing.


Risk based testing is all about "Test" the risks in the object that you are testing. That is simpler while saying. In reality - Risk based testing involves

1. A methodical approach for identifying risks
2. Design the tests to explore the risks - know more about identified risks - As quickly as possible
3. Precipitate the failures resulting from the risks - execute tests - As quickly as possible
4. Explore other possible (unknown in the beginning) risks - As quickly as possible
5. Repeat steps 1..4 until all risks evaporate to the reasonable degree OR you run out of time/budget.


If you claim to have mastered Risk based testing technique - check you have following -

1. A proven method of identifying the risks in the product domain - most of these proven ones have a strong theoretical or empirical back ground ( mathematical/statistical or similar). A method you can rely on.
2. Test design methodology - that is matured enough to design tests to explore and precipitate this risks
3. An exit Criteria to know when are "reasonably done with"
4. Knowledge of "Risks" associated with this Risk based ( ironical isn't?) approach to testing.


There seem to be two broad categories of Risk based testing methodologies

1. Exhaustive Risk modeling and analysis - those with involved mathematical and statistically models one the lines of those used for "Reliability Engineering"

2. Heuristic Risk based Testing - rather simplistic one. James Bach has written an excellent article on this topic. As described in the article this is one of the approaches that can be adopted ( in combination with other "formal" techniques) when you have resource and time crunch. He cautions that the whole concept of "Heuristic" could be a fallible one and more of provisional and plausible. It tends to explore the solution.

I happen to discuss this issue with others in Google group on software testing, here. There are some very interesting and thought provoking ideas from Andy Tinkham in this thread.


Read following lines carefully - before you finish reading this post and tell me what do you think …
1. Testing is more about finding important problems that threaten the value of the product - issues that bug someone who matters in the context
2. Risk is a problem that might happen - higher the probability of Risk - higher is the probability of failure.
3. Should not all testing be risk based? Why would one want to test where there is no risk? When non risk based testing would be appropriate ….

Shrini

Group Test Manager - Role Description

I happen get mail from a head hunter, asking me if I am interested in a "Group Test Manager" position in a reputed Company. When I looked at the job description, what surprised me is that - not even once the word "Testing" appeared in the profile of the job that required to manage a group of test managers. I am not sure this "profile acrobatics" is from newbie head hunter or from a HR trainee who was given the task of drafting job requirements for Group Test manager.

What is the problem here? What if the word "testing" does not appear? What is big deal if I confuse "Testing" and "Quality assurence" or "SQA" or "Production Mangement" or "Software process Facilitation"? The danger of such job posting is that it is loose-loose game. As there will be expectation miss match from both sides. It will be good if the expectations are settled down before the person starts the job. If that does not happen then - chances are that either company or the candidate will start looking for alternatives within months in the new job. Think of the cost of recruitment, training, relocation and other indirect costs that involved in bringing a person onboard - especially at such senior level. If, in the beginning, the hiring manager and HR - fail to put right effort in drafting job requirements and matching title, there would be dark clouds over the future.

Notice that in this job description - the pre-requiresites (Excellent Communication, Interpersonal, Analytical Skills) mentioned last and under "Needless to mention" category. It is unfortunate that these essential requirements for the job are taken for granted - is this the case?

====================================================
Position - Group test manager
Time line - very urgent
Experience 9 to 14 years
BRIEF PROFILE: Must have
Degree in Computer Science or related field or equivalent work experience and
Minimum of 8 years of related experience in SQA and/or Production Management required Combination of equivalent training and related work experience with experience managing technical teams.
Good software engineering process facilitation skills.
Excellent problem resolution, judgment, and decision making skills required.
Strong mentoring, team building, and counseling skills on software engineering and career issues.·
Excellent written and oral communication skills required.
Willingness to travel internationally on a regular basis (at least twice a year)
Nice to have· Overseas experience
Work experience in offshore and outsourced environment
Full SDLC experience· ITSM or Six-Sigma knowledge
Needless to mention: Excellent Communication, Interpersonal, Analytical Skills are pre-requisites.
============================================

Here is another example of vague and ambiguous job description. This is from very reputed internet MNC.

Need QA Engineers with 2-6 experience in following

- Experience in Linux, Solaris and Unix
- Experience in White box
- Experience in Seibel and QTP

================================================

May God bless - the company who requires people with above mentioned skills, Head hunter who hunts for the people, Interviewer who does the interviews and finally the poor candidate -applying for this job (like someone who enters automobile "spare parts" shop asking for medicines)

Finally, if you are somebody who is currently drafting job descriptions for your test positions - Need help in reviewing and fine tuning them. Feel free to contact me at shrinik@gmail.com for my services in this regard.

Shrini

10 Classic Reasons why Test Automation projects Fail ...

Yes, this is the title of the paper that will be talking about at STeP IN Summit 2006 International Conference on Software testing at Bangalore, INDIA.

Follow this link for the conference details

http://www.stepin.qsitglobal.com/speakers.html

Be there ....

Shrini

www.crazytestingideas.com ?

Hold on !!! this is not the name of my new website nor this is another site I want you to rush and have look at. Then, What the heck is this? Read on ....

At times, when you as tester are frustrated - as you were not able to find bugs - might have done something funky, totally out of box, or outrageously creative or totally un-imaginable and that led you to some interesting bugs? When was it that jumped out of the chair and yelled “Eureka!!!! I found a BUG - Look at this?"

Something on the lines of Shoe testing as James Bach mentions in his Rapid testing class?

One such I was trying to do was similar to Shoe testing. I was testing an application screen that lists some items where the number and status was constantly changing - something on the lines of Flight arrival and departure status in an Airport Gaint Display screen. There was button for "Refresh" I just used a pen top or a clip to stick "enter key" and so that stays pressed - before doing tha I started monitors like Filemon, Task manager at performance Tab ( CPU usage and memory) and left for lunch.

When I came back after an hour so, application had died with some weird message. Looked like a memory leak issue. No would have imagined that application would be used in this way. Through this I came to know about some interesting about the application - very unusual type of testing but some times powerful. Developer might react to this and say “No would ever do this?” “This is not a typical user scenario”. I say – "Look this weird test exposed some unknown behavior about this application which none new till today. I leave it up to you whether you want to fix it or not”

Have you come across such crazy test ideas? Do share –

How about hosting a website - http://www.crazytestingideas.com/ ? Volunteers please? OR Venture capitalist to fund this new company that sells crazy testing ideas on web :-):-)

Shrini

Vision of an expert Tester?

Have you heard any skilled tester claiming this Vision? "I can test any thing, under any conditions and under any time frame". I do. Read this another "James Gems" article. Very cleverly he supports this tough goal with qualifiers. What would you do as expert tester if asked to test a nuclear power plant in 30 minutes? How will you react to test "IBM's Deep blue Chess software" in 2 hours? James gives few hints in this article as how to handle such seemingly impossible testing goals.

BTW, thinking of it, all testers should slowly march towards developing ability of "Testing anything, anywhere, under any time frame". Yes with your own qualifiers. It is a tough goal but worth pursuing – in my opinion.

Thank you James - for setting all skilled testers to this journey towards MOON - every tester is invited.

Shrini