“This troublesome user acceptance testing got to be easier. It is really a pain” said of my colleague quoting an IT manager responsible for release management and user acceptance testing. Major problem according the IT manager was getting the “users” to do their job – testing a defect fix, feature enhancement or a new release. Users are always busy with their “day job” and for most of them doing testing is a low priority activity. I have heard many similar stories regarding this important aspect of “end user” responsibility.
In traditional IT organizations, user acceptance testing is a formal stage in the testing life cycle where business users will test the proposed changes to production software applications. IT organization that delivers the software to meet the needs of business users, require a (formal) approval of proposed changes that come in terms of defect fixes, enhancements and new releases. Since it is business users that ask for changes for their existing and fund the development/testing work – they will have the final say on “acceptance” of proposed changes to production applications.
Primary purpose of user acceptance testing is check (cursory- final) proposed changes to the production software by those users that asked for the changes so that any last minutes changes and surprises are avoided. User acceptance test is the last gate before software goes Live so that it provides last opportunity for all involved (both IT and business) to make corrections if required. Depending upon the nature of business supported, nature of software application, nature of changes proposed UAT may last for few days to few weeks. Software that fails in UAT is typically assumed to be insufficiently tested and is variably returned to IT for fixes and more testing.
The problem of UAT
Business users who hold the responsibility of approving software updates to production systems, typically are not testers and as such do not come with testing skills. Depending upon the nature of software updates, IT team will ask business users to carry out testing to check the features of new system. Most of the user acceptance testing tends to be repetitive and that is what drives “real users” crazy. For development team (IT), their work is not complete until the piece of code is user tested and accepted. Hence they literally chase user group to perform UAT with users complaining about the time crunch and their “important” business work getting impacted. This creates a situation where both parties (IT and business) want UAT to be somehow completed so that each can carry out their business as usual. Hence UAT often gets “fixed” with the consent of both parties having stakes in the activity.
Forms of fixing UAT
Between Business and IT, UAT can be fixed in number of ways – some of these are reasonably business driven models given the constraints.
1. UAT will be performed by the members from IT team where business only reviews the results of the test. If the results are OK then UAT is ought to have been completed
2. UAT will be performed by a third party, such as someone from support staff. Business will review the results and accept the proposed changes if results are OK
3. Business will provide a prescribed set of test scripts that can either be used by anyone IT staff or support staff. Results will be verified by the Business.
4. UAT will be done like a demo to the users where IT staff will execute some pre-approved test scenarios related to proposed changes.
5. IT team will train the business in new features proposed to be introduced. Business users after training, test (use) the proposed software and accept the software
Why fixing user acceptance testing is bad thing?
Why bother to UAT after all? For IT, more often than not, it is a formality to be completed before they push the code to production.
In my opinion, it is the spirit and purpose of UAT that gets compromised. Typically, in spite of all best efforts, the depth and frequency of interactions between IT and business throughout the project remains low. When business users do not participate with full spirit in UAT, lots of things go unnoticed into production. This might results users (non participating ones especially) getting surprised when they see the product.
What has been your experience? Do you bother if you see your UAT is fixed?
Shrini
9 comments:
Nice post.
Its quite true that end users do not fullfil the responsibility of testing before the application is gonna live in production,
I agree with your suggestions :)
Keep it up :)
Some comments:
Major problem according the IT manager was getting the “users” to do their job – testing a defect fix, feature enhancement or a new release.
Is that their job? Or is that something extra that they've been asked to do on top of their job? My money is on the latter; that they are not allocated time for the testing task, nor are there rewards or incentives for doing it... but there is punishment—explicit or implicit—for not doing it. So when you hear that kind of comment from an IT manager, responding with some questions about incentives and rewards and motivation might be useful. Ask how the users' day jobs change when they're asked to test; ask if their workload for other tasks is reduced; and so forth.
Another pair of questions to ask: "What happens when the users report a problem?" and "What doesn't happen when they report a problem?" I'm guessing that little or nothing happens when they report a problem, and thus they're trained into believing that what they do doesn't matter.
Software that fails in UAT is typically assumed to be insufficiently tested and is variably returned to IT for fixes and more testing. Invariably? In my experience (and we've both done work at the same organization where I've observed these things in action),
- the actual users of the application aren't consulted during planning and development; people in higher-level or management positions sometimes are;
- user experience and workflow are not really studied at all, either as theory or in direct observation of people doing the work;
- in both programming and testing, there's a strong focus on functional correctness and none on usability;
- when problems are discovered the end users are offered workarounds instead of fixes;
- problems that would be "too hard to fix" are deemed as "training issues", "cosmetic problems", or "nice to haves"--and then deferred;
- the software is deployed anyway, as-is.
Most of the user acceptance testing tends to be repetitive and that is what drives “real users” crazy.
How is it repetitive? In what ways is it repetitive? Must it be repetitive? What would happen if it weren't repetitive, and were more aligned with the work that the end users actually do?
---Michael B.
Hi Shrini,
A well written post, you have covered all points/problems we face in our day to day business. In addition to 5 points listed, I would like to add something from my experience; I faced situations where UAT goes simultaneously with Regression, sounds strange. The reason is, either end users of the product do not have time or there is a time constraint and pressure to release product ASAP.
Can you further elaborate on the ways to deal with these problems, of getting UAT out of FIX.
One thing I could think of getting Users (end users) involved at each and every stage of Application development....would it help?
Hi Shirini,
Nice post. This is the ground reality that we are facing in our project. There were some releases which was in UAT for more than a month.
On top of this, there are cases where the "Buisness" demanded hot fixes much earlier that the whole release itself. So we again had to go back to the older version of the code and work on these 'hot fixes'. Right now we have as much as 3 code bases on which we are parlley working. :(
I dont think there is nothing much we can do about it.
What has been your experience? Do you bother if you see your UAT is fixed?
To put this another notation i see UAT as a signoff from the customer to say that the code can be deployed in the production. I go with your suggestions but what we practice is that, we define an acceptance criteria with the business people on a mutual agreement during the kick off.
This acceptance criteria is what now we concentrate on whose results are then verified by the business giving them more space to take care of their work.
I try to insist this, the less effort we require from these geniuses, the more happy they are and the more happy they are the more happy we are. No one wants to loose their grin nowadays and I believe it is better to resolve these before we kick off the project.
In my point of view, user acceptance testing is useful if it is carried out in an environment that closely resembles the real world or production environment.
User acceptance testing is also known as end user testing. After the application is fully developed the User Acceptance Testing is done. Different types of testing such as Unit testing, Integrity testing, System testings are done before the User acceptance testing. As these tests are done before the UAT (User Acceptance Testing), most of the bugs are fixed during previous levels of testing.
I want to share a website address http://www.successful-quality-assurance.com/types-of-software-testing.html which is dedicated to software quality assurance.
very nice post,I am learning the testing n its very helpful for me
It is very interesting for me to read the post. Thank you for it. I like such topics and anything connected to them. I definitely want to read a bit more soon.
Actually,good post. thx
Post a Comment