Software Testing.....'X'perts, THE BUG HUNTERS!!!

ISEB vs ISTQB

2010-02-05T02:19:00.001-08:00

ISEB vs. ISQTB: which one to choose?

This article aims to compare the ISEB & ISTQB certification schemes for Software Testing and provide important information for those who already hold the ISEB or

ISTQB Foundation certification to choose the next step. The structure of the schemes outlined here demonstrates the link and differences between them.

ISEB Overview

The Information Systems Examinations Board (ISEB) is a division of the British Computer Society. It was created in 1990 from the Systems Analysis Examinations

Board that was set up in 1967. It administers examinations and issues certificates in a variety of subjects in the field of information systems engineering. In 2008,

ISEB made changes to the Practitioner Certificate based on feedback from candidates and training providers.

There were two key problems that ISEB aimed to solve with the changes:

• Firstly, that the transition from the foundation level exam to an all encompassing practitioner level with one exam was far too steep, and it was

recommended that the gap be bridged in order to assist candidates with their progression

• Secondly, it became apparent that as testers tend to specialise in either the management of testing or in test analysis but seldom both disciplines, many

candidates who had taken the exam recommended in their feedback that it would be beneficial to develop separate exams for each.

Therefore, in order to cater for candidate needs, ISEB has redeveloped the original qualification to address all of these issues. ISEB now offer four qualifications,

one foundation, one general and two role based.

1. Foundation.

2. Intermediate Certificate in Software Testing.

3. Practitioner Certificates:

§ Test Management

§ Test Analysis

Diploma in Software Testing (Will be awarded if you pass both the above Practitioner Certificates).

The ISTQB Overview

The International Software Testing Qualifications Board (ISTQB) was founded in Edinburgh in November 2002. ISTQB’s role is to support a single, universally

accepted, international qualification scheme, by providing the core syllabi and by setting guidelines for accreditation and examination for national boards.

It is a fast growing and democratic organisation made up of an assembly of 45 member boards with each board representing major software and system

developing countries and regions.

ISTQB has developed a new Advanced Level Certification which is seen as an alternative to the ISEB Practitioner level. A key difference from the ISEB model is that

there is no intermediary level required from Foundation to Advanced level. Following on from the foundation level, the ISTQB Advanced Level Certification has one

syllabus which has been divided into three core areas.

1. Foundation.

2. Advanced Level

• Test Manager

• Test Analysis

• Technical Test Analysis

Expert Level (Will be awarded if you pass all the above advance level Certificates).

The exam for each area is in multiple choice formats, with a certificate being awarded for each area. An additional ‘Advanced Level Testing Professional’ certificate

is awarded for gaining all three.

ISEB & ISTQB Certifications

The foundation levels of both certification schemes are transferable as ISEB has adopted the syllabus from the ISTQB Foundation level. This situation is positive as

it provides the candidate with alternative options (i.e. once a candidate has completed the Foundation level, he/she can choose to start either certification). Once

the Foundation level is complete the differences between the certifications become clear. ISEB has an Intermediate level preceding its practitioner level, while the

ISTQB syllabus does not.

Once a student passes either Foundation level exam, a decision must be made regarding which "certification path" to pursue. For a UK-based software tester, the

logical decision is to go down the ISEB accreditation route. In this scenario, the next level would be the ISEB Intermediate, followed by either the ISEB Practitioner

in Test Analysis or the ISEB Practitioner in Test Management certifications.

Anyone outside of the UK should pursue the ISTQB path, as this certification has a global appeal and is formally recognised in 45 countries. There is only one

additional ISTQB certification, and that is the ISTQB Advanced certification. The ISTQB Advanced level is equivalent to the ISEB Practitioner certifications (test

analysis or test management). There is no certification akin to the ISEB Intermediate level for ISTQB certifications.

The ISTQB Advanced level exam is split into 3 modules: Technical Test Analyst, Test Analyst and Test Manager. Each has its own exam and accreditation. Students

passing all 3 modules earn the highly desirable ‘Advanced Level Testing Professional' certification.

Testing professionals, having achieved the Full Advanced Level Testing professional level will then be able to proceed to the ISTQB Expert level, which aligns with

the ISEB Practitioner Diploma level.

What’s the difference?

How does the ISTQB certification differ from ISEB?

ISEB certifications are issued by the International Systems Examinations Board (ISEB) which is an UK based international examination body founded in order to

raise the standard of competence and performance of people working in IT. It has higher recognition in UK.

ISTQB certifications are issued by the International Software Testing Qualifications Board (ISTQB) which is an UK based institution where software professionals

from all over the world work together to define standardized contents for further education in Software Testing. Hence it has a wider international recognition.

ISTQB currently do not recognise the new revised ISEB Practitioner Certificates as equivalent to the full Expert Level Testing Professional certificate. This means

that the expert level in ISTQB cannot be reached through the ISEB Practitioner qualification but instead requires ISTQB Advanced as a pre-requisite.

ISEB and ISTQB covers similar topics but examine to different levels of competency. The ISTQB Advanced level exams will examine up to Level K4 which is the same

level of competency as the ISEB Intermediate Certificate in Software Testing. Whereas the ISEB Practitioner qualifications examine competence at a higher level,

up to Level K6, adding real value for candidates who wish to progress in their chosen specialism.

The structure of the two schemes is also different, mainly in the fact that ISEB has three levels and ISTQB has two only levels. Both schemes begin with a

foundation level certification following which ISEB offers an intermediate level certification before the final level. ISTQB does not have the intermediate level and

offers the final level directly after the foundation level certification.

Recommendation

Well I support the ISTQB Advanced Syllabus rather than the ISEB Practitioner course. The reasoning behind this recommendation is as follows:

• ISTQB is the international standard for the software testing qualifications and was developed by an international team. Hence there is a lot of international

support for this qualification.

• International clients prefer the ISTQB qualification.

• ISTQB qualification is formally recognized in around 45 countries where the ISTQB has a constituent organisation.

• The ISTQB Advanced syllabus includes the ISEB Intermediate syllabus also.

• The ISTQB Syllabus, through Learning Objectives and detailed description makes it clear what should be taught and what will be examined.

• The route to attaining ISTQB Advanced level in one of the core areas i.e. Test Management, Test Analysis or Technical Test Analysis is more straightforward

and direct than the ISEB route, involving only one exam and fewer training days.

• Multiple choice exam questions enable non-English speaking candidates to complete the exams more easily.

Conclusion:

ISTQB certifications have only been brought in as an alternative as they are specifically designed for global acknowledgement, 41 countries have so far recognised

it as a valid certificate. However there is no known extra skill gained through ISTQB that is not provided by ISEB, which in plain English means that one will not lose

out if they complete an ISEB instead of an ISTQB certification.

Examples of Specification Rules for Design

2009-06-16T05:55:00.000-07:00

AR1: Cost Detail: The architecture must be specified in enough detail to at least permit correct order of magnitude impact estimation for costing.
AR2: Cost Estimates: Estimates must be made and included as to the order of magnitude of all critical costs of the architecture (particularly for those resources that are budgeted requirements).
AR3: Performance Detail: The architecture specification must include enough detail to allow order of magnitude correct estimation of the specification’s impact on ALL specified performance
goal levels (that is, all the work capacity, qualities, and savings requirements).
AR4: Performance Estimates: Estimates will be included for the impacts on ALL the critical performance requirements, at correct order of magnitude.
AR5: Background Detail for Estimate: Each impact estimate must be supported by:the factual experiential evidence for the estimate, the source of these facts, the uncertainty boundaries/error margins (±%), and a credibility rating (on a 0 to 1.0 scale). These data ideally will be presented on an impact estimation table.
AR6: Additional Data: The architectural specification must include additional specification as detailed in the current architecture specification template. This will include Stakeholders, Detailed Design Reference (if any), QC level, Review Approval level, Risks, Issues, and Dependencies.

Software Testing Best Practice Award

2008-02-23T03:14:00.000-08:00

The International Institute for Software Testing is
giving away FIFTY days of free training.

Up to TEN companies may win. Each company will get Five
days of free training.

Apply for the IIST's Software Testing Best Practice Award
at http://www.iist.org/bestpractice and receive the following benefits:

**** Get your company featured as an Award Winning Company
In all IIST publicity channels and press releases
**** Tell everyone how great your test process is
**** Get 5 days of free training at the International
Testing Certification Super Week to be held in
Las Vegas, NV, November 26-30, 2007
Chicago, IL, March 24-28, 2008
See details at http://iist.org/superweek
**** Get your company and your test team recognized and identified
as an Award Winning Team during these remarkable events

The International Testing Certification Super Week will be
held in the following cities:

Las Vegas, NV, November 26-30, 2007
Chicago, IL, March 24-28, 2008

During these events, IIST will offer 25 Full day
in-depth courses taught by Leading Industry Experts
in Software Testing & Quality.

**** Register by September 30th, and save 20% (Las Vegas)
See details at: http://www.iist.org/stpw/lasvegas/index.php

**** Register by December 31st, 2007 and save 30% (Chicago)
See details at: http://www.iist.org/stpw/chicago08/index.php

2008-02-23T03:12:00.000-08:00

Reporting bugs - A how to guide

One of my colleague forwarded me a good link regarding bug reporting for web application testing; worth a quick read.

“Reporting bugs - a how-to guide”

an article from edgeofmyseat.com

http://www.edgeofmyseat.com/articles/2007/07/08/reporting-bugs/

Full Text of the article

-----------------------------------------------------

When working with a developer or team of developers on an application – whether you are a designer working with developers or an end client hiring developers – you all want the same end result, a slick and bug free application. During the testing process of any application it is likely that some bugs or issues will show up and this article aims to explain how to report bugs and problems effectively so that your developers don’t need to spend time working out what the problem is before being able to fix it. This helps to ensure that projects stay on budget and that developers are spending their time adding features to the application rather than trying to get enough details to be able to reproduce and fix issues.

“It’s just not working!”

When you find a problem, it is very tempting to just fire off an email and presume that the developer will immediately be able to see the problem too. However, by taking a few minutes to describe the problem you have encountered accurately you can prevent any confusion occurring as to what the problem is and save both your time and the developer’s as she won’t need to get back to you to find out what actually happened, or spend a long time trying to reproduce the issue.

A good report

A good bug report tells your developer three vital things:

What you expected to happen

What actually happened

What you did/were doing when it happened

What you expected to happen

There are two kinds of ‘bugs’, the first is where something breaks – you see an error message, your uploaded data disappears, you submit a form and the change isn’t saved. These bugs are generally pretty easy to report and identify as all your developer needs is to know exactly what you were doing or inputting at the time and they should be able to reproduce and fix the issue.

The second kind of bug is where the application doesn’t function as you expected. This might be because the developer has misinterpreted part of the specification or it could be that what you expect just isn’t how something can work. In this case the developer believes that it is working fine – and in fact it is ‘working’ even if it is incorrect. If your bug report is that the feature is broken, the developer may then spend time looking for some error in this part of the application when what they need to realize is that it isn’t working as you expected. By giving the information about what you expected to happen the developer can think ‘ah … you wanted it to do x and it is doing y’ and a resolution can be sorted out quickly.

What actually happened

What actually happened is very rarely ‘nothing’ yet bug reports often contain the phrase, ‘nothing happened’. If what happened was ‘nothing’ in terms of the intended result then explain that in a few more words, for example, if you clicked the submit button on a form and it didn’t submit and go onto the next page you could say,

“The form didn’t submit – it just remained on the same page.”

Or perhaps the form submitted and a blank page displayed,

“After submitting the form a blank page loaded.”

If an error message displays on the screen then include that in the report. Just copy and paste the error message.

If you use Internet Explorer then your browser may not display the error message generated by the server, instead showing a generic error page. You can ensure the IE displays the real error message by going to Tools > Internet Options > Advanced. Then scroll down to the browsing section and uncheck ‘Show Friendly HTTP error messages’.

What you were doing when it happened

Your developer wants to know this information – not because they want to tell you that you were doing something wrong, but because it is highly likely that the bug occurs only when a certain path of actions is followed, or when a certain type of data is entered. The more information you can give your developer the easier it will be for them to reproduce the problem you saw and fix it. Things you should include:

The steps taken
List exactly what you did, in the order you did it if possible. If you can go back and try the same steps again and the problem happens again that is great – note down exactly how you made the problem occur. Your developer will be pleased as you have just saved her time trying to reproduce the issue. Even if you can’t reproduce it, no-one is going to doubt that the problem happened – just describe as much as you can remember how you got to the broken point.

Any data you were entering
If the problem happened after you added some data to a form, include the data with the bug report. If you were uploading something such as an image into the application then include that too.

It may also be helpful to copy and paste the URL out of the address bar of the browser so the developer knows exactly which page you were on at the time.

The browser and operating system you were using at the time
With web applications problems may only be occurring in one browser. Let your developer know exactly what you are using – including the version number - so they can create the same environment to test the problem.

Effective bug reporting can make a huge difference in how quickly problems can be resolved, and prevent frustration on both sides of the process. Including the above information, even if it doesn’t seem relevant, will be appreciated by the developer. You don’t need to write an essay, just a few clear lines explaining the key information of:

what you expected to happen

what actually happened and,

what you did/were doing when it happened.

This will be enough to isolate all but the most complicated of issues, and once an issue can be reproduced it is well on its way to being fixed.

Requirements Testing

2008-02-23T03:10:00.000-08:00

Testing software is an integral part of building a system. However, if the software is based on inaccurate requirements, then despite well written code, the software will be unsatisfactory. Most of the defects in a system can be traced back to wrong, missing, vague or incomplete requirements.

Requirements seem to be ephemeral. They flit in and out of projects, they are capricious, intractable, unpredictable and sometimes invisible. When gathering requirements we are searching for all of the criteria for a system's success. We throw out a net and try to capture all these criteria.

The Quality Gateway

As soon as we have a single requirement in our net we can start testing. The aim is to trap requirements-related defects as early as they can be identified. We prevent incorrect requirements from being incorporated in the design and implementation where they will be more difficult and expensive to find and correct.

To pass through the quality gateway and be included in the requirements specification, a requirement must pass a number of tests. These tests are concerned with ensuring that the requirements are accurate, and do not cause problems by being unsuitable for the design and implementation stages later in the project.

Make The Requirement Measurable

In his work on specifying the requirements for buildings, Christopher Alexander describes setting up a quality measure for each requirement.

"The idea is for each requirement to have a quality measure that makes it possible to divide all solutions to the requirement into two classes: those for which we agree that they fit the requirement and those for which we agree that they do not fit the requirement."

In other words, if we specify a quality measure for a requirement, we mean that any solution that meets this measure will be acceptable. Of course it is also true to say that any solution that does not meet the measure will not be acceptable.

The quality measures will be used to test the new system against the requirements. The remainder of this paper describes how to arrive at a quality measure that is acceptable to all the stakeholders.

Quantifiable Requirements

Consider a requirement that says "The system must respond quickly to customer enquiries". First we need to find a property of this requirement that provides us with a scale for measurement within the context. Let's say that we agree that we will measure the response using minutes. To find the quality measure we ask: "under what circumstances would the system fail to meet this requirement?" The stakeholders review the context of the system and decide that they would consider it a failure if a customer has to wait longer than three minutes for a response to his enquiry. Thus "three minutes" becomes the quality measure for this requirement.

Any solution to the requirement is tested against the quality measure. If the solution makes a customer wait for longer than three minutes then it does not fit the requirement. So far so good: we have defined a quantifiable quality measure. But specifying the quality measure is not always so straightforward. What about requirements that do not have an obvious scale?

Non-quantifiable Requirements

Suppose a requirement is "The automated interfaces of the system must be easy to learn". There is no obvious measurement scale for "easy to learn". However if we investigate the meaning of the requirement within the particular context, we can set communicable limits for measuring the requirement.

Again we can make use of the question: "What is considered a failure to meet this requirement?" Perhaps the stakeholders agree that there will often be novice users, and the stakeholders want novices to be productive within half an hour. We can define the quality measure to say "a novice user must be able to learn to successfully complete a customer order transaction within 30 minutes of first using the system". This becomes a quality measure provided a group of experts within this context is able to test whether the solution does or does not meet the requirement.

An attempt to define the quality measure for a requirement helps to rationalise fuzzy requirements. Something like "the system must provide good value" is an example of a requirement that everyone would agree with, but each person has his own meaning. By investigating the scale that must be used to measure "good value" we identify the diverse meanings.

Sometimes by causing the stakeholders to think about the requirement we can define an agreed quality measure. In other cases we discover that there is no agreement on a quality measure. Then we substitute this vague requirement with several requirements, each with its own quality measure.

Requirements Test 1

Does each requirement have a quality measure that can be used to test whether any solution meets the requirement?

By adding a quality measure to each requirement we have made the requirement visible. This is the first step to defining all the criteria for measuring the goodness of the solution. Now let's look at other aspects of the requirement that we can test before deciding to include it in the requirements specification.

Requirements Test 2

Does the specification contain a definition of the meaning of every essential subject matter term within the specification?

When the allowable values for each of the attributes are defined it provides data that can be used to test the implementation.

Requirements Test 3

Is every reference to a defined term consistent with its definition?

Requirements Test 4

Is the context of the requirements wide enough to cover everything we need to understand?

Requirements Test 5

Have we asked the stakeholders about conscious, unconscious and undreamed of requirements?

Requirements Test 5 (enlarged)

Have we asked the stakeholders about conscious, unconscious and undreamed of requirements? Can you show that a modelling effort has taken place to discover the unconscious requirements? Can you demonstrate that brainstorming or similar efforts taken place to find the undreamed of requirements?

Requirements Test 6

Is every requirement in the specification relevant to this system?

Requirements Test 7

Does the specification contain solutions posturing as requirements?

Requirements Test 8

Is the stakeholder value defined for each requirement?

Requirements Test 9

Is each requirement uniquely identifiable?

Requirements Test 10

Is each requirement tagged to all parts of the system where it is used? For any change to requirements, can you identify all parts of the system where this change has an effect?

Conclusions

The requirements specification must contain all the requirements that are to be solved by our system. The specification should objectively specify everything our system must do and the conditions under which it must perform. Management of the number and complexity of the requirements is one part of the task.

The most challenging aspect of requirements gathering is communicating with the people who are supplying the requirements. If we have a consistent way of recording requirements we make it possible for the stakeholders to participate in the requirements process. As soon as we make a requirement visible we can start testing it. and asking the stakeholders detailed questions. We can apply a variety of tests to ensure that each requirement is relevant, and that everyone has the same understanding of its meaning. We can ask the stakeholders to define the relative value of requirements. We can define a quality measure for each requirement, and we can use that quality measure to test the eventual solutions.

Testing starts at the beginning of the project, not at the end of the coding. We apply tests to assure the quality of the requirements. Then the later stages of the project can concentrate on testing for good design and good code. The advantages of this approach are that we minimise expensive rework by minimising requirements-related defects that could have been discovered, or prevented, early in the project's life.

References:

An Early Start to Testing: How to Test Requirements
Suzanne Robertson

Testac, a proprietary test design tool

2008-02-06T06:15:00.000-08:00

Testac, a proprietary test design workbench which hallmarks Maveric’s expertise in Testing. Testac promises to significantly improve test coverage and crash timelines by over 40%, in business critical software and systems. Approximately USD one million has been invested into the development of Testac over the last three years. Testac-led services will fuel Maveric’s growth and increase their revenues by 30% in the next three years. It is now available in three different suites namely, Testac Retail Banking (core, consumer and mortgage), Testac Wholesale Banking (Core and corporate lending) and Testac Islamic Banking (retail and corporate).

Testac is a pioneering test design workbench that significantly enhances the quality of Maveric’s testing services and compresses test cases to a fraction of 10. Testac brings rigour to the test design phase, focussing on “What to test” and “How much to test”. It is the only knowledge-embedded test design workbench that forces a detailed systematic view of the application under test by following business processes and transaction flows to generate scenarios and cases. Moving forward, Maveric will continue to invest on this robust platform to further enrich their User Acceptance Testing (UAT) services.

“According to an IDC report on the software testing industry, the market opportunity for the Indian offshore testing companies is currently $2 billion, and is estimated to rise to $8 billion in 2008,” said Mr. Ranga Reddy, Chief Executive Officer, Maveric Systems. “Testac has led to a paradigm shift from a pure service model to an IP based services model thereby making us the only Indian company to have its proprietary intellectual property in the testing domain. The financial services domain expertise of our testing team coupled with our knowledge embedded design workbench have resulted in significant benefits to our customers,” he added.

Over the last six months, Maveric has deployed Testac for their clients in the middle-east including The Dubai Islamic Bank, Bank Muscat and Emirates Bank. Maveric has recently been awarded the coveted ‘Product Innovation Award’ by Frost & Sullivan in the software testing category at the Sixth ‘India ICT Awards’ hosted in Mumbai in Dec 2007. At present, Maveric plans to target corporates in the banking, finance, insurance and BPO sectors in India and UK.

Maveric has strongly advocated the “Model based testing” approach to effectively test end-to-end business scenarios and Testac is based on this concept. A functional framework of the related business segments is embedded into the test design workbench. Some of the key benefits of Testac include

1. Helps deliver better test coverage

2. Significant reduction in timelines upto 30%

3. Minimises test duplication

4. Carries deep domain knowledge

5. Permits prioritisation during execution

Testac Workbench has three integral components viz.,

1. A Business Modeler that holds pre-defined standard models for specific verticals. Models capture business definitions and rules at the product, transaction and business process levels.

2. A Design Generator that applies powerful algorithms and design principles to the business model to come up with optimised test cases.

3. An integrator that connects the workbench to upstream and downstream tools in defect and test management

Maveric as an organisation constantly innovates and adapts to the ever-changing landscape of testing domain. This new test design workbench will support Maveric’s commitment to provide an industry-leading software-testing platform for the future.

About Maveric Systems Ltd

Maveric Systems Ltd. is one of India’s top three independent software-testing companies since the year 2000. Over the past eight years, Maveric has been pioneering the outsourcing model in software testing. Maveric has successfully moved up the value chain in software testing by taking clients away from a resource augmentation model to provide value added IP-led services that significantly enhance quality of applications deployed by clients. As a leading pure-play, independent software testing company, Maveric today has multiple delivery locations in India, UK, Middle East and USA. Maveric’s impressive client list includes a wide array of leading software product companies, system integrators and financial institutions. With employee strength of 500, Maveric has been growing rapidly over the past five years,

For more information on Maveric Systems, please visit www.maveric-systems.com

Delivery Management

2008-01-29T03:37:00.000-08:00

Definition - Delivery Management: the endeavor management task of properly managing the delivery of work products and services

Responsibilities - The typical responsibilities of Delivery Management are to:

Ensure that all deliverable work products and services are delivered as required by the binding legal agreements.

Ensure that all deliverable work products and services are delivered according to the budget and schedule.

Eliminate risks due to poor delivery.

Preconditions - Delivery management typically may begin when the following preconditions hold:

Either the: Endeavor has started or Center exists.
The management team is adequately:
Staffed.
Trained in delivery management.

Completion Criteria - Delivery management is typically complete when the following postconditions hold:

Either the:
Endeavor has been completed or the Center has been retired.
All deliverable work products and services have been successfully delivered to (and accepted by) the customer organization.
All commercial packages have been successfully delivered by the vendor organization(s) to the development organization.
All components have been successfully delivered by the subcontractor organization(s) to the development organization.

Steps - Delivery management typically involves the management team performing the following steps in an iterative, incremental, parallel, timeboxed, and ongoing manner:

Work with the development organization and customer organization to ensure that the deliverable work products and services are delivered as agreed.
Work with the vendor organization(s) to ensure that they successfully deliver acceptable commercial packages to the development organization.
Work with the subcontractor organization(s) to ensure that they successfully deliver acceptable components to the development organization.
Track deliveries against schedules and milestones.

Techniques - Delivery management typically involves the following techniques:
Close collaboration with external organizations.
Correspondance.
Phone conversations.
Face to face conversations.
Meetings.

Work Products - Delivery management results in the following work products:

Management Set:
Delivery Statement

Guidelines - Delivery management is critical because most endeavors fail to deliver all that was promised on time and within budget.
Relationship management and communication management are critical to successful delivery management.

Happy Testing !!!

Staging environment ???

2008-01-29T03:22:00.000-08:00

Definition
The staging environment is any development environment that is primarily used to stage tested applications prior to their deployment to the production environment.

Objectives - The typical objectives of the staging environment are to provide a separate environment for the:

Content management team to incorporate approved content prior to publication to the content management production environment.

User experience team to perform system usability testing.

Customer representatives to evaluate the completeness, quality, and general status of the tested application prior to deployment.

Benefits - The typical benefits of the staging environment include:

Content problems can be found prior to publication in the production environment.

System usability testing can be performed in its own realistic environment without slowing down system and launch testing.

Customer representatives can evaluate the tested application without slowing down testing and deployment.

Hope this explains you about staging environment mail me if u need some more clarification on this.

Happy Testing !!!

Cricket Vs Software testing :)

2008-01-16T05:06:00.000-08:00

Cricket is a bat-and-ball sport contested by two teams, usually of eleven players each. Let us see if we can correlate the sports cricket with that of software testing! I am going to identify and list out few points that seem similar between software testing and cricket:

1. (Un)Predictability: Experts say, ‘unpredictability of cricket is its greatest charm’! It is very difficult to predict a win or loss before the last ball is bowled. And it is the unpredictability, which makes cricket so interesting. Teams might look strong or weak on a piece of pen and paper. But in cricket, unpredictability reins, not the statistics.

Coming to software testing, there could be testers who believe that it is easy and straightforward to expect some fixed output after executing a series of test steps, but I only wish; testing software was as simple as that! Echoing some experts in software testing: trying to predict the result of a test in terms of PASS or FAIL criteria can be one of those dangerous traps in software testing world, where a tester can shoot at his own feet! Irrespective of the number of test scripts (either manual test cases or automated test scripts) a tester has written, until the tester gets the application module to test, nothing can be told for sure about the state of the application and its behavior. Unpredictability is one of those things that make software testing so much fun.

2. Skills: Cricket is a game where only skillful players can make their team a winner. I am not sure about other countries, but if you have ever been to India, chances are high that you might have seen cricket being played on a street behind your hotel room! They say, cricket is a fever here in India. Here people are so obsessed with the game that they not only play and watch cricket but also eat, sleep and even drink cricket! I have played fair amount of cricket in my school and college days. You might have too. But the reason, why players like Sachin Tendulkar, Brian Lara and Ricky Ponting are considered as one of the finest cricketers and not us, lies in the cricketing skills they possess.

Likewise, in software testing too, it is the testing skill that differentiates a good tester from a mediocre one. The one, who has the better testing skills in his arsenal, can find more important defects quickly in the software that he is testing. Learning, practicing and applying are three golden rules to acquire any skill. With determination and strong will power, nothing is impossible to learn. Fortunately, software testing is no exception!

3. Game Planning (Knowing the Opponent): Professional cricket is all about knowing the strengths and weaknesses of the opponent team and devising a game plan in order to combat their strengths and to exploit their weaknesses.

In software testing, knowing the testing mission is the first step in determining the goal of the testing effort. Without being clear about the goal, it might turn out fatal to go about testing straightaway. Once the tester is clear about the testing mission, then he can analyze his chances of success or failure depending on the availability and expertise of the resource in hand and the complexity of the testing problem. e.g. imagine a situation where the tester has to test the application to find out it’s robustness to guard against hackers and other malicious users. Knowledge of a fair deal of details about the level of attack that can be attempted against the application, can give the tester a better chance to plan out a strategy to emulate the attack and to test how the application guards against it.

4. Handling Pressure: Winning a game of cricket is all about handling the pressure well. The team that is able to handle the tremendous amount of pressure of the game finally wins the game.

For those who think or have been told that software testing is an easy job and can be done by every Tom Dick and Harry, let me warn you, you have been terribly misguided! Testing is a career, which demands lot of intellectuality and stability of mind to work under variety of pressures (like technical pressure, pressure due to workload, managerial pressure, pressure of deadline, pressure arising from the nature of the job etc). As testers, the basic requirement of our job demands us to deliver bad news (presence of defects, buggy modules that fail miserably on testability grounds etc) to different stakeholders (the programmers, management staffs, clients) of the product under development. Nobody likes to hear bad news. Unfortunately, gone are the days when messengers were not hanged just because they brought in a bad news to the king! Hence, unless the tester is quite good at handling the pressure arising as a byproduct of his work, and is not so good at being diplomatic, he might find it tough to carry on with his job for long. On the other hand, the tester who has got the capability to handle the pressure till the end, has every chance of winning the Testing World Cup!

5. Adaptability: If you have been watching cricket for sometime now, then you must be experienced enough to know that whenever a team visits a different continent for playing cricket, it often finds it difficult to play up to its usual standards, of course unless we are talking about a team like Australia. Most Asian teams find it difficult to play down under and the vice versa. And the reason lies mainly in the difference in the pitch condition in the different continents. Soil texture, clay quality, humidity, temperature, amount of grass, dust etc can affect the behavior of a cricket pitch. The players who can adapt themselves quickly with the new environment can gain an edge in the game.

Just as no two cricket pitches can be exactly identical, no two AUTs (Application Under Test) can be same. Hence, I often get surprised when I see testers trying to clean themselves off testing assignments with excuses such as:

» “I have not tested anything like this before!”
» “This platform is completely new to me!”
» “I am new to this technology. I can not test it!”
» “I am a manual tester. I can not use this test automation tool!”
» “I need the domain knowledge before I can test this application!”
» “Hey, you don’t have any base documents (URS, SRS, BRS blah blah…)! How can I start testing your application?”

To me, this simply translates as a bad attitude of a tester, who is not ready to adapt and more importantly not ready to learn something new. For such testers, I have just one thing to say:

"Don’t forget that, even big and powerful animals like the Dinosaurs and the Wooly Mammoth became extinct just because they FAILED to adapt to their changing surrounding environments; we are just testers!"

When I say so, I DON’T mean that I am a great tester who has been excellent at adaptability throughout his testing career. On the contrary, I have been through several such instances when I was stupid enough to oppose such changes and was in fact, hostile to proposed changes. And don't be too surprised when I tell you that, some of the excuses that I have mentioned above were from my own mouth! However, I have learnt my lesson of adaptability in the hard, bitter way. So I just wanted to warn others who are new and passionate about testing but might commit the same mistake of lack of adaptability that I have committed in my earlier years of testing career.

6. Patience: Cricket is a game of patience. At times, a batsman might be finding it tough to steal a run due to the sharp bowling attack, tough ground conditions etc. But if the batsman has got enough temperament and patience, he might soon find it easy to not only steal a single but also score boundaries against the same bowler under same pitch conditions. Similarly with patience and consistent good bowling, a bowler could turn around the fate of a match by picking up quick wickets.

Similarly, a lot of patience is required while testing an application. There might be days when nothing seemed to work right for you and you would end up without catching a single defect in the whole testing session. There might be times when you would find yourself hitting your head on the keyboard while trying to find ways to reproduce a hard-to-reproduce defect. But you ask any expert in software testing, and he would suggest you to have patience. Putting in extra effort and having patience can help you come out of such dry days of testing.

7. Team Game: Cricket is a team game. Here the team wins the match, which displays excellent exhibition of team play. And so is software testing. All the team members of a testing team can’t be with equal skill sets and with equal level of expertise. Having a testing team with members with different specialization can add to the versatility of the team’s overall performance. I do understand that there can be testing geniuses, who are jack of all testing trades and are all-rounders when it comes to testing. But finding such a tester can be a tough task for any test manager. So as a workaround a good test manager might look forward to build a team which has testers specializing in different aspects of testing (functionality testing, performance/stress testing, database testing, usability testing, automation engineers, risk-based testing etc).

8. Lack of Concentration can Doom you: Cricket requires a high level of concentration. As a batsman you loose your concentration and you could loose your wicket. As a bowler you loose your concentration and you could be hit for boundaries all over the ground! As a fielder you loose your concentration and you could drop a catch (remember! catches win matches!) or give extra runs by mis-fielding.

Coming back to testing, imagine a tester who misses to identify a defect that just happens in front of his eyes, as he was looking for some other defect at that moment. Experts call this phenomenon as inattentional blindness. This can also happen if your mind is too tired due to continuous testing and you start loosing concentration, and in turn, start loosing defects!

9. Wearing safety Helmets and Testers: Batsmen (and even the close-in fielders) wear helmets to safeguard their head from the fast coming deliveries from the bowler. In case of software testing, unfortunately, the testers are like the safety helmet! They act as the representative of the client/end users of the project. And act as the last line of defense of the client/end users against the possible ways in which the software might fail. So as testers, start feeling proud of your profession, as you are safeguarding somebody by taking the hits yourself.

10. Learning Lessons from Failures: Even world champions like Australia was once defeated in a cricket match against the under-dogs like Bangladesh. Success and failures are two integral part of any game. The team that wins today can loose the next match and vice versa. But what differentiates a world champion from an average team is the extent to which they learn their lessons from their failures.

On a similar note, a software testing project can go terribly bad due to various reasons. I have witnessed and have been part of few such testing projects that went nowhere nearer where they were intended to go! But each experience, good or bad, has some lessons in it. A good tester tries to learn lessons from his past failures to make sure they are not repeated in future. The failure might not have been as a result of a mistake on the part of the tester, but still, there might be some lessons in that failure that could help the tester in becoming better at his testing in future.

If you have reached this paragraph after patiently reading through my post, then I must appreciate you for your patience. You have certainly shown a desirable skill of a good tester – patience. :) I would like to hear your ideas about my analogy between cricket and software testing. Can you think of more such points, which can link them together? Let me know your ideas by commenting. Howzat!!!

Happy Testing…

“If the software product has some defects, can it still be called a quality product?”

2008-01-16T04:59:00.000-08:00

This is a question I was asked during a recent conversation with one of my blog readers (who is also a software tester). It appeared that the tester friend was asked this question in some testing interview and the interviewer had expected a simple YES/NO type answer! Unfortunately the tester, who was unsure about the answer, also expected a similar (YES/NO) answer from me for the above question. To me, this actually seems like a paradoxical question, which can be answered only after breaking it down further into few individual questions, which can throw more light on the context of the subject. But before I attempt to answer this question, let me assume few definitions of the keywords used in the question.

Software Product: A software product is typically a single application or suite of applications built by a software company to be used by *many* customers, businesses or consumers. The mass-market notion differs from custom software built for the use of a single customer by software development firms. – Wikipedia

Defects: A software defect (or "bug") is an error, flaw, mistake, failure, or fault in a computer program that prevents it from behaving as intended (e.g., producing an incorrect result). Most software defects arise from mistakes and errors made by people in either a program's source code or its design, and a few can also be caused by compilers producing incorrect code. – Wikipedia

Quality: Quality is value to some person, who matters. – Gerry M. Weinberg

Coming back to the original question, when someone asks, “if the software product has some defects, can it still be called a quality product”, I think a possible answer is: “I can’t tell you for sure unless you are ready to answer few of my questions first”!

If you are a software tester, I am almost sure that you are yet to see software, which does not have any defects in it (even after thorough testing). Even if the testers are not able to identify any significant amount of defects with further testing activity, still no tester with a stable brain on his shoulder would ever dare to state that “the software is defect-free after the testing effort”! In that sense, all software is with some defect (either known defects or yet-to-be-found defects). Since no software is without defect, what is the point in asking something like: "if the software product has some defect …"? What do you think?

I am not sure if the interviewer actually wanted to ask: “if the software product has some *known* defects *after it’s release*, can it still be called a quality product?”

This sounds to me like a more meaningful and tactable question which can be attempted for answering! As a software tester, I understand that there can be situations when a product is known to have some unfixed defects and yet the management can decide to release the product. This can particularly happen when,

a) The known defects are of low severity and are less devastating. Even if the end-user encounters such defects, it is speculated that the defects would not cause much trouble/nuisance to the end-user.
b) These are corner case defects and the chance of the end-user being hit by such defects is less.
c) The stakeholders of the project don’t feel that these defects are of much importance!

Whatever might be the reason behind the decision to release the product with those known defects, I am not sure if we can surely tell that the shipped product is of high quality or not, simply on the basis of the defects shipped with the product! Echoing some experts in software testing, quality of software is much more than presence or absence of defects. Imagine a software product, which is shipped with a belief that it does not have any known defects and is quite stable. But wait! What if the end-user does not like it, as it is difficult to learn and operate? What if, it is difficult to train others how to use the software? In a single sentence, what if the product fails severely from usability aspect? Think of a video game, which is very robust (does not crash easily) and lacks any visible defect, and yet the persons who play this game don’t find it interesting enough! Can we still call it a quality software/video game? I guess not!

Coming back to our original question, even though a software product has defects, it could still be called a quality product. At the same time, even though a software product has lesser known-defects, it might not be of good quality! Quality of software is multi-dimensional and can't be decided by the presence or absence of defects alone! Hence, it might be little tricky to answer the above question, without being clear on the understanding of quality by the particular organization. As a tester, do you think you can answer the above question? I am eager to hear your views on the subject.

Happy Testing…

Testers can get and share information with developers, to improve collaboration and make better programs with fewer struggles.

2007-12-07T12:00:00.001-08:00

While I'm not sure that the common saying "Testers and developers think differently" is accurate, I think it's safe to say that testers and developers certainly have different motivators for their work—and, more importantly, different pressures and perspectives that guide the directions their work takes. Testers are motivated by management to find and report problems within the system as quickly as possible, while developers are motivated to complete code as quickly and accurately as possible in order to move on to the next problem.

Somewhere in this jumble of speed versus quality, someone should be motivating testers to find better, more meaningful bugs, but I rarely see that motivation happening. And someone should be motivating developers to improve the quality of the code they create or modify, but that prompting often takes a backseat to "More code, faster."

Because of these different motivations of testers and developers, and the time pressures that everyone on the team is under, communication suffers. It's easy for a tester to think of a simple test that the developer should have run to find a particular problem before releasing the code—but the tester doesn't appreciate the pressures imposed on the developer. And it's easy for the developer to be dismayed by an overwhelming number of low-value or meaningless defects being reported—but the developer doesn't appreciate the expectations and metrics against which management measures the tester.

As more defects are submitted, developers have less time to work on them, and communication may break down. Members of the project team commonly start to rely on short, assumptive descriptions and comments entered into a defect-tracking system as the fastest and most effective means of communication. Little time is spent face to face, where the most effective communication can take place and dialogues can offer insights to both the tester and the developer.

In this article, I'll discuss how recent projects used the following simple techniques to fix problems faster and improve communication between testers and developers:

· Sharing test scripts between teams

· Distributing the ability to execute smoke tests

· Performing runtime analysis together

· Using log files to isolate problems

· Using defect-tracking systems effectively

· Speaking face to face

Not all of these techniques added "face time" to our project communications, but all of them helped everyone involved to gain a better understanding of the pressures and constraints under which both developers and testers work.

Share Test Scripts Between Teams

Problems encountered by test scripts can be lengthy or difficult to reproduce. In past projects, I've submitted defects that took hours to reproduce due to the sequence of events in the scripts being executed. More often than not, this setup requirement made it impossible for the developer to reproduce the issue based on the information in the log. If you make all of your test scripts available to the entire team, however, you give developers the ability to look at the script code, look at the script logs, re-run the scripts and watch them execute, re-run them in local environments with debug information written to logs, or re-run them in conjunction with other tools.

In addition to sharing the test scripts, provide the development team with a remote automated test-script execution box. Most enterprise tools allow for distributed test execution. If you provide one of your test lab machines to run your scripts, developers can execute the tests they need while simultaneously using their own computers to keep developing. This technique allows developers to work on your problem; without the testing box, the developers might not be able to run a full test due to time and equipment constraints. To make this strategy most effective, reference your scripts in the submitted defect list. The development team can run the scripts without checking with you first, removing a manual step.

Sharing test scripts with developers also enables everyone on the team to use the same tools used to develop the scripts. When team members use the same tools, a likely side-effect is developers taking the time to offer improvements to the scripts. After running one of my scripts, a developer once told me that he already had a unit-test script that did something similar "behind the GUI." Together we reviewed both scripts, and ultimately we transferred the data from my regression script to work with his unit-test script. His unit-test script executed in a fraction of the time of my regression script, and the results were easier to read. The more feedback from developers you get on your scripts, the more powerful they'll become.

In addition, the more you collaborate with developers, the more likely they'll be to fix your problem; after all, it may have been their optimization that found the bug. By distributing the ability to execute any of your scripts, you increase communication between developers and testers.

Testers vs. Developers

2007-12-07T11:58:00.000-08:00

Assertions can be derived as soon as a specification exists, however in this code-based methodology, assertion localization does not occur until after code testability is measured. Thus since testability measurements must be performed before the assertions are derived, this process cannot be performed until later in the software development life-cycle.

However, knowing that, for example, variable a needs to be tested after assignment statement 100, does not indicate what constraint the assertion should be testing for (to determine whether a problem with a has occurred). Hence our methodology has only addressed one part of the oracle/assertion problem, specifically placement, not derivation. Someday we would like to link this tool up with a formal specification language, and automatically generate the assertions, but that remains a research goal for the future.

As mentioned earlier, testers are likely to be incapable of deriving correct assertions, while developers are more likely to derive assertions that mimic the semantics of the code already there. If the developer does not understand the specification, their assertions will be incorrect. That is to say that if a developer's code is faulty, their assertions will almost certainly be faulty.

This brings us to our final recommendation for how to team developers and testers in a partnership to strategically derive and embed assertions. Let the testers find where assertions are needed, and leave it to the developers to determine what the assertions should be. Note that this is different than the case where the developer determines both where to place the assertions and what they should be. Here, the developer is being forced to derive assertions that the tester needs for places in the code where the developer might not be as sure as to what the assertion should be. If this happens, it forces developers to dig deeper into the code and requirements than they might have already done. This plays a similar role as code inspections, except that the person digging into the code is also the person that is likely to have written the code. Although this is not a foolproof solution, it is the best recommendation that we can provide at this time. Having developers spend more time comparing their previous understanding of the code to the existing requirements can only serve to improve the code's quality. Even if the assertion that the developer derives does not detect an error, the fact the the developer is forced to derive the assertion will increase the likelihood that the developers themselves find errors, because they are forced to get more familiar with less familiar internal computations in the code.

CMM-Capability Maturity Model

2007-11-25T15:35:00.001-08:00

Capability Maturity Model:The Capability Maturity Model (CMM) is a methodology used to develop and refine an organization's software development process. The model describes a five-level evolutionary path of increasingly organized and systematically more mature processes. CMM was developed and is promoted by the Software Engineering Institute (SEI), a research and development center sponsored by the U.S. Department of Defense (DoD). SEI was founded in 1984 to address software engineering issues and, in a broad sense, to advance software engineering methodologies. More specifically, SEI was established to optimize the process of developing, acquiring, and maintaining heavily software-reliant systems for the DoD. Because the processes involved are equally applicable to the software industry as a whole, SEI advocates industry-wide adoption of the CMM.
The CMM is similar to ISO 9001, one of the ISO 9000 series of standards specified by the International Organization for Standardization (ISO). The ISO 9000 standards specify an effective quality system for manufacturing and service industries; ISO 9001 deals specifically with software development and maintenance. The main difference between the two systems lies in their respective purposes: ISO 9001 specifies a minimal acceptable quality level for software processes, while the CMM establishes a framework for continuous process improvement and is more explicit than the ISO standard in defining the means to be employed to that end.

What is CMM (SEI Capability Maturity Model)? The Capability Maturity Model for Software (CMM) is a framework that describes the key elements of an effective software process. There are CMMs for non software processes as well, such as Business Process Management (BPM). The CMM describes an evolutionary improvement path from an ad hoc, immature process to a mature, disciplined process. The CMM covers practices for planning, engineering, and managing software development and maintenance. When followed, these key practices improve the ability of organizations to meet goals for cost, schedule, functionality, and product quality. The CMM establishes a yardstick against which it is possible to judge, in a repeatable way, the maturity of an organization's software process and compare it to the state of the practice of the industry. The CMM can also be used by an organization to plan improvements to its software process. It also reflects the needs of individuals performing software process, improvement, software process assessments, or software capability evaluations; is documented; and is publicly available.

Testing Life Cycle Team Structure

2007-11-25T15:32:00.001-08:00

An effective testing team includes a mixture of members who has Testing expertise/Tools expertise.
Database expertise/Domain/Technology expertise.
Consultants/End users.
The testing team must be properly structured, with defined roles and responsibilities that allow the testers to perform their function with minimal overlap.
There should not be any certainty regarding which team member should perform which duties.
The test manager will be facilitating any resources required for the testing team.

Iteration Model

2007-11-25T15:31:00.003-08:00

Spiral Model

2007-11-25T15:31:00.001-08:00

Prototype Model

2007-11-25T15:30:00.003-08:00

Waterfall Model

2007-11-25T15:30:00.001-08:00

The waterfall Model is an engineering model designed to be applied to the development of software.The idea is the following: there are different stages to the development and the outputs of the First Stage "Flow" into the second stage and these outputs "flow" into the third stage and so on.there are Usually five stages in this model of software Development.

Stages of Water Fall Model

Requirement analyis and Planninng:- In This stage the requirements of the "To be developed Software" are established.These are usually the services it will provide,its constraints and Goalsof the Software.Once theae are Established they Have to be defined such a way that are usable in the next Stage.This Stage is often Preludes by a Feasibility or a Feasible study is included in this stage.the Feasibility study includes Questions like;Should we develop the Software,what are the alternatives? It could be called the conception of a software project and might be seen as the very begining of the life cycle.

V & V PROCESS MODEL :

2007-11-25T15:29:00.001-08:00

V&V Model is Verification & Validation Model.In This Model We work simultaneously Development and Testing.In this Model One V for Verification and one For Validation first 'V' we follow SDLC(software Development Life Cycle) and Second 'V' we follow STLC-(Software Testing Life Cycle).

Testing normally done in a large system in 2 parts. The functional verification and validation against the Requirement specification and Performance evaluation against the indicated requirements.
Testing activity is involved right from the beginning of the project.
Use of V&V process model increases the rate of success in a project development company to deliver the application on time and increases the cost effectiveness.

Testing Related Activities During Requirement Phase

Creation and finalization of testing template.
Creation of test plan and test strategy .
Capturing Acceptance criteria and preparation of acceptance test plan.
Capturing Performance Criteria of the software requirements.

Testing activities in Design Phase

Develop test cases to ensure that product is on par with Requirement Specification document.
Verify Test Cases & Test Scripts by peer reviews.
Preparation of traceability matrix from system requirements.

Testing activities in Unit Testing Phase

Unit test is done for validating the product with respect to client requirements.
Testing can be in multiple rounds.
Defects found during system test should be logged in to defect tracking system for the purpose of resolving and tracking.
Test logs and defects are captured and maintained.
Review of all test documents.

Testing activities in Integration Testing Phase

This testing is done in parallel with integration of various applications or components.
Testing the product with its external and internal interfaces without using drivers and stubs.
Incremental approach while integrating the interfaces.

Performance testing

This is done to validate the performance criteria of the product/ application. This is non-functional testing.

Business Cycle testing

This refers to end to end testing of real life like business scenarios.

Testing activities during Release phase

Acceptance testing is conducted at the customer location.
Resolves all defects reported by the customer during Acceptance testing.
Conduct Root Cause Analysis (RCA) for those defects reported by customer during acceptance testing.

Software Testing Dictionary

2007-11-25T15:25:00.001-08:00

Acceptance Testing
Testing conducted to enable a user/customer to determine whether to accept a software product. Normally performed to validate the software meets a set of agreed acceptance criteria.

Accessibility Testing
Verifying a product is accessible to the people having disabilities (deaf, blind, mentally disabled etc.).

Ad Hoc Testing
A testing phase where the tester tries to 'break' the system by randomly trying the system's functionality. Can include negative testing as well. See also Monkey Testing.

Agile Testing
Testing practice for projects using agile methodologies, treating development as the customer of testing and emphasizing a test-first design paradigm. See also Test Driven Development.

Automated Software Quality (ASQ)
The use of software tools, such as automated testing tools, to improve software quality.

Basis Path Testing
A white box test case design technique that uses the algorithmic flow of the program to design tests.

Basis Set
The set of tests derived using basis path testing.

Beta Testing
Testing of a re-release of a software product conducted by customers.

Binary Portability Testing
Testing an executable application for portability across system platforms and environments, usually for conformation to an ABI specification.

Black Box Testing
Testing based on an analysis of the specification of a piece of software without reference to its internal workings. The goal is to test how well the component conforms to the published requirements for the component.

Bottom Up Testing
An approach to integration testing where the lowest level components are tested first, then used to facilitate the testing of higher level components. The process is repeated until the component at the top of the hierarchy is tested.

Boundary Testing
Test which focus on the boundary or limit conditions of the software being tested. (Some of these tests are stress tests).

Bug
A fault in a program which causes the program to perform in an unintended or unanticipated manner.

Boundary Value Analysis
BVA is similar to Equivalence Partitioning but focuses on "corner cases" or values that are usually out of range as defined by the specification. his means that if a function expects all values in range of negative 100 to positive 1000, test inputs would include negative 101 and positive 1001.

Branch Testing
Testing in which all branches in the program source code are tested at least once.

Breadth Testing
A test suite that exercises the full functionality of a product but does not test features in detail.

CAST
Computer Aided Software Testing.

Capture/Replay Tool
A test tool that records test input as it is sent to the software under test. The input cases stored can then be used to reproduce the test at a later time. Most commonly applied to GUI test tools.

CMM
The Capability Maturity Model for Software (CMM or SW-CMM) is a model for judging the maturity of the software processes of an organization and for identifying the key practices that are required to increase the maturity of these processes.

Cause Effect Graph
A graphical representation of inputs and the associated outputs effects which can be used to design test cases.

Code Complete
Phase of development where functionality is implemented in entirety; bug fixes are all that are left. All functions found in the Functional Specifications have been implemented.

Code Coverage
An analysis method that determines which parts of the software have been executed (covered) by the test case suite and which parts have not been executed and therefore may require additional attention.

Code Inspection
A formal testing technique where the programmer reviews source code with a group who ask questions analyzing the program logic, analyzing the code with respect to a checklist of historically common programming errors, and analyzing its compliance with coding standards.

Code Walkthrough
A formal testing technique where source code is traced by a group with a small set of test cases, while the state of program variables is manually monitored, to analyze the programmer's logic and assumptions.

Compatibility Testing
Testing whether software is compatible with other elements of a system with which it should operate, e.g. browsers, Operating Systems, or hardware.

Concurrency Testing
Multi-user testing geared towards determining the effects of accessing the same application code, module or database records. Identifies and measures the level of locking, deadlocking and use of single-threaded code and locking semaphores.

Conformance Testing
The process of testing that an implementation conforms to the specification on which it is based. Usually applied to testing conformance to a formal standard.

Context Driven Testing
The context-driven school of software testing is flavor of Agile Testing that advocates continuous and creative evaluation of testing opportunities in light of the potential information revealed and the value of that information to the organization right now.

Conversion Testing
Testing of programs or procedures used to convert data from existing systems for use in replacement systems.

Cyclomatic Complexity
A measure of the logical complexity of an algorithm, used in white-box testing.

Data Driven Testing
Testing in which the action of a test case is parameterized by externally defined data values, maintained as a file or spreadsheet. A common technique in Automated Testing.

Defect
Nonconformance to requirements or functional / program specification

Dependency Testing
Examines an application's requirements for pre-existing software, initial states and configuration in order to maintain proper functionality.

Depth Testing
A test that exercises a feature of a product in full detail.

Dynamic Testing
Testing software through executing it. See also Static Testing.

Endurance Testing
Checks for memory leaks or other problems that may occur with prolonged execution.

End-to-End testing
Testing a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate.

Equivalence Class
A portion of a component's input or output domains for which the component's behavior is assumed to be the same from the component's specification.

Equivalence Partitioning
A test case design technique for a component in which test cases are designed to execute representatives from equivalence classes.

Exhaustive Testing
Testing which covers all combinations of input values and preconditions for an element of the software under test.

Functional Decomposition
A technique used during planning, analysis and design; creates a functional hierarchy for the software.

Functional Specification
A document that describes in detail the characteristics of the product with regard to its intended features.

Functional Testing
· Testing the features and operational behavior of a product to ensure they correspond to its specifications.
· Testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions.

Gorilla Testing
Testing one particular module,functionality heavily.

Gray Box Testing
A combination of Black Box and White Box testing methodologies: testing a piece of software against its specification but using some knowledge of its internal workings.

High Order Tests
Black-box tests conducted once the software has been integrated.

Independent Test Group (ITG)
A group of people whose primary responsibility is software testing.

Inspection
A group review quality improvement process for written material. It consists of two aspects; product (document itself) improvement and process improvement (of both document production and inspection).

Integration Testing
Testing of combined parts of an application to determine if they function together correctly. Usually performed after unit and functional testing. This type of testing is especially relevant to client/server and distributed systems.

Installation Testing
Confirms that the application under test recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions.

Localization Testing
This term refers to making software specifically designed for a specific locality.

Loop Testing
A white box testing technique that exercises program loops.

Monkey Testing
Testing a system or an Application on the fly, i.e just few tests here and there to ensure the system or an application does not crash out.

Negative Testing
Testing aimed at showing software does not work. Also known as "test to fail".

Path Testing
Testing in which all paths in the program source code are tested at least once.

Performance Testing
Testing conducted to evaluate the compliance of a system or component with specified performance requirements. Often this is performed using an automated test tool to simulate large number of users.

Positive Testing
Testing aimed at showing software works. Also known as "test to pass".

Quality Assurance
All those planned or systematic actions necessary to provide adequate confidence that a product or service is of the type and quality needed and expected by the customer.

Quality Audit
A systematic and independent examination to determine whether quality activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives.

Quality Circle
A group of individuals with related interests that meet at regular intervals to consider problems or other matters related to the quality of outputs of a process and to the correction of problems or to the improvement of quality.

Quality Control
The operational techniques and the activities used to fulfill and verify requirements of quality.

Ramp Testing
Continuously raising an input signal until the system breaks down.

Recovery Testing
Confirms that the program recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions.

Regression Testing
Retesting a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.

Release Candidate
A pre-release version, which contains the desired functionality of the final version, but which needs to be tested for bugs (which ideally should be removed before the final version is released).

Sanity Testing
Brief test of major functional elements of a piece of software to determine if its basically operational.

Scalability Testing
Performance testing focused on ensuring the application under test gracefully handles increases in work load.

Security Testing
Testing which confirms that the program can restrict access to authorized personnel and that the authorized personnel can access the functions available to their security level.

Smoke Testing
A quick-and-dirty test that the major functions of a piece of software work. Originated in the hardware testing practice of turning on a new piece of hardware for the first time and considering it a success if it does not catch on fire.

Soak Testing
Running a system at high load for a prolonged period of time. For example, running several times more transactions in an entire day (or night) than would be expected in a busy day, to identify and performance problems that appear after a large number of transactions have been executed.

Software Testing
A set of activities conducted with the intent of finding errors in software.

Static Analysis
Analysis of a program carried out without executing the program.

Static Analyzer
A tool that carries out static analysis.

Static Testing
Analysis of a program carried out without executing the program.

Storage Testing
Testing that verifies the program under test stores data files in the correct directories and that it reserves sufficient space to prevent unexpected termination resulting from lack of space. This is external storage as opposed to internal storage.

Stress Testing
Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements to determine the load under which it fails and how. Often this is performance testing using a very high level of simulated load.

Structural Testing
Testing based on an analysis of internal workings and structure of a piece of software.

System Testing
Testing that attempts to discover defects that are properties of the entire system rather than of its individual components.

Testability
The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met.

Testing
· The process of exercising software to verify that it satisfies specified requirements and to detect errors.
· The process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs), and to evaluate the features of the software item (Ref. IEEE Std 829).
· The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.

Test Bed
An execution environment configured for testing. May consist of specific hardware, OS, network topology, configuration of the product under test, other application or system software, etc. The Test Plan for a project should enumerated the test beds(s) to be used.

Test Case
· Test Case is a commonly used term for a specific test. This is usually the smallest unit of testing. A Test Case will consist of information such as requirements testing, test steps, verification steps, prerequisites, outputs, test environment, etc.
· A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.

Test Driven Development
Testing methodology associated with Agile Programming in which every chunk of code is covered by unit tests, which must all pass all the time, in an effort to eliminate unit-level and regression bugs during development. Practitioners of TDD write a lot of tests, i.e. an equal number of lines of test code to the size of the production code.

Test Driver
A program or test tool used to execute a tests. Also known as a Test Harness.

Test Environment
The hardware and software environment in which tests will be run, and any other software with which the software under test interacts when under test including stubs and test drivers.

Test First Design
Test-first design is one of the mandatory practices of Extreme Programming (XP).It requires that programmers do not write any production code until they have first written a unit test.

Test Harness
A program or test tool used to execute a tests.

Test Plan
A document describing the scope, approach, resources, and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.

Test Procedure
A document providing detailed instructions for the execution of one or more test cases.

Test Script
Commonly used to refer to the instructions for a particular test that will be carried out by an automated test tool.

Test Specification
A document specifying the test approach for a software feature or combination or features and the inputs, predicted results and execution conditions for the associated tests.

Test Suite
A collection of tests used to validate the behavior of a product. The scope of a Test Suite varies from organization to organization. There may be several Test Suites for a particular product for example. In most cases however a Test Suite is a high level concept, grouping together hundreds or thousands of tests related by what they are intended to test.

Test Tools
Computer programs used in the testing of a system, a component of the system, or its documentation.

Thread Testing
A variation of top-down testing where the progressive integration of components follows the implementation of subsets of the requirements, as opposed to the integration of components by successively lower levels.

Top Down Testing
An approach to integration testing where the component at the top of the component hierarchy is tested first, with lower level components being simulated by stubs. Tested components are then used to test lower level components. The process is repeated until the lowest level components have been tested.

Total Quality Management
A company commitment to develop a process that achieves high quality product and customer satisfaction.

Traceability Matrix
A document showing the relationship between Test Requirements and Test Cases.

Usability Testing
Testing the ease with which users can learn and use a product.

Use Case
The specification of tests that are conducted from the end-user perspective. Use cases tend to focus on operating software as an end-user would conduct their day-to-day activities.

Unit Testing
Testing of individual software components.

Validation
The process of evaluating software at the end of the software development process to ensure compliance with software requirements. The techniques for validation is testing, inspection and reviewing.

Verification
The process of determining whether of not the products of a given phase of the software development cycle meet the implementation steps and can be traced to the incoming objectives established during the previous phase. The techniques for verification are testing, inspection and reviewing.

Volume Testing
Testing which confirms that any values that may become large over time (such as accumulated counts, logs, and data files), can be accommodated by the program and will not cause the program to stop working or degrade its operation in any manner.

Walkthrough
A review of requirements, designs or code characterized by the author of the material under review guiding the progression of the review.

White Box Testing
Testing based on an analysis of internal workings and structure of a piece of software. Includes techniques such as Branch Testing and Path Testing. Also known as Structural Testing and Glass Box Testing.

Workflow Testing
Scripted end-to-end testing which duplicates specific workflows which are expected to be utilized by the end-user.

Functional Testing Vs Non-Functional Testing

2007-11-25T15:21:00.001-08:00

Functional Testing: Testing the application against business requirements. Functional testing is done using the functional specifications provided by the client or by using the design specifications like use cases provided by the design team.

Functional Testing covers:

Unit Testing
Smoke testing / Sanity testing
Integration Testing (Top Down,Bottom up Testing)
Interface & Usability Testing
System Testing
Regression Testing
Pre User Acceptance Testing(Alpha & Beta)
User Acceptance Testing
White Box & Black Box Testing
Globalization & LocalizationTesting

Non-Functional Testing: Testing the application against client's and performance requirement. Non-Functioning testing is done based on the requirements and test scenarios defined by the client.

Non-Functional Testing covers:

Load and Performance Testing
Ergonomics Testing
Stress & Volume Testing
Compatibility & Migration Testing
Data Conversion Testing
Security / Penetration Testing
Operational Readiness Testing
Installation Testing
Security Testing (ApplicationSecurity, Network Security, System Security)

Software Testing Metrics

2007-11-25T15:20:00.000-08:00

- Cost of finding a defect in testing (CFDT):

Total effort spent on testing / defects found in testing

Note: Total time spent on testing including time to create, review, rework, execute the test cases and record the defects. This should not include time spent in fixing the defects.

- Test Case Adequacy: This defines the number of actual test cases created vs estimated test cases at the end of test case preparation phase. It is calculated as

No: of actual test cases / No: of test cases estimated

- Test Case Effectiveness: This defines the effectiveness of test cases which is measured in number of defects found in testing without using the test cases. It is calculated as

No: of defects detected using test cases*100/Total no: of defects detected

- Effort Variance:

{(Actual Efforts-Estimated Efforts) /Estimated Efforts} *100

- Schedule Variance:

{(Actual Duration - Estimated Duration)/Estimated Duration} *100

- Schedule Slippage: Slippage is defined as the amount of time a task has been delayed from its original baseline schedule. The slippage is the difference between the scheduled start or finish date for a task and the baseline start or finish date. It is calculated as

((Actual End date - Estimated End date) / (Planned End Date – Planned Start Date) * 100

- Rework Effort Ratio:

{(Actual rework efforts spent in that phase / Total actual efforts spent in that phase)} * 100

- Review Effort Ratio:

(Actual review effort spent in that phase / Total actual efforts spent in that phase) * 100

- Requirements Stability Index:

{1 - (Total No. of changes /No of initial requirements)}

- Requirements Creep:

(Total No. of requirements added/No of initial requirements) * 100

Weighted Defect Density:

WDD = (5*Count of fatal defects)+(3*Count of Major defects)+(1*Count of minor defects)

Note: Here the Values 5, 3, 1 correspond to severities as mentioned below:

Fatal - 5
Major - 3
Minor - 1

Load Runner - Interview Questions

2007-11-25T15:19:00.001-08:00

1. What is load testing?
Load testing is to test that if the application works fine with the loads that result from large number of simultaneous users, transactions and to determine weather it can handle peak usage periods.

2. What is Performance testing?
Timing for both read and update transactions should be gathered to determine whether system functions are being performed in an acceptable timeframe. This should be done standalone and then in a multi user environment to determine the effect of multiple transactions on the timing of a single transaction.

3. What is LoadRunner?
LoadRunner works by creating virtual users who take the place of real users operating client software, such as sending requests using the HTTP protocol to IIS or Apache web servers. Requests from many virtual user clients are generated by Load Generators in order to create a load on various servers under test these load generator agents are started and stopped by Mercury's Controller program. The Controller controls load test runs based on Scenarios invoking compiled Scripts and associated Run-time Settings. Scripts are crafted using Mercury's "Virtual user script Generator" (named "V U Gen"), It generates C-language script code to be executed by virtual users by capturing network traffic between Internet application clients and servers. With Java clients, VuGen captures calls by hooking within the client JVM. During runs, the status of each machine is monitored by the Controller. At the end of each run, the Controller combines its monitoring logs with logs obtained from load generators, and makes them available to the "Analysis" program, which can then create run result reports and graphs for Microsoft Word, Crystal Reports, or an HTML webpage browser.

Each HTML report page generated by Analysis includes a link to results in a text file which Microsoft Excel can open to perform additional analysis. Errors during each run are stored in a database file which can be read by Microsoft Access.

4. What is Virtual Users?
Unlike a WinRunner workstation which emulates a single user's use of a client, LoadRunner can emulate thousands of Virtual Users.Load generators are controlled by VuGen scripts which issue non-GUI API calls using the same protocols as the client under test. But WinRunner GUI Vusers emulate keystrokes, mouse clicks, and other User Interface actions on the client being tested. Only one GUI user can run from a machine unless LoadRunner Terminal Services Manager manages remote machines with Terminal Server Agent enabled and logged into a Terminal Services Client session.

During run-time, threaded users share a common memory pool. So threading supports more Vusers per load generator.

The Status of Vusers on all load generators start from "Running", then go to "Ready" after going through the init section of the script. Vusers are "Finished" in passed or failed end status. Vusers are automatically "Stopped" when the Load Generator is overloaded.

To use Web Services Monitors for SOAP and XML, a separate license is needed, and vUsers require the Web Services add-in installed with Feature Pack (FP1). No additional license is needed for standard web (HTTP) server monitors Apache, IIS, and Netscape.

5. Explain the Load testing process?
Step 1: Planning the test. Here, we develop a clearly defined test plan to ensure the test scenarios we develop will accomplish load-testing objectives.

Step 2: Creating Vusers. Here, we create Vuser scripts that contain tasks performed by each Vuser, tasks performed by Vusers as a whole, and tasks measured as transactions.

Step 3: Creating the scenario. A scenario describes the events that occur during a testing session. It includes a list of machines, scripts, and Vusers that run during the scenario. We create scenarios using LoadRunner Controller. We can create manual scenarios as well as goal-oriented scenarios. In manual scenarios, we define the number of Vusers, the load generator machines, and percentage of Vusers to be assigned to each script. For web tests, we may create a goal-oriented scenario where we define the goal that our test has to achieve. LoadRunner automatically builds a scenario for us.

Step 4: Running the scenario. We emulate load on the server by instructing multiple Vusers to perform tasks simultaneously. Before the testing, we set the scenario configuration and scheduling. We can run the entire scenario, Vuser groups, or individual Vusers.

Step 5: Monitoring the scenario. We monitor scenario execution using the LoadRunner online runtime, transaction, system resource, Web resource, Web server resource, Web application server resource, database server resource, network delay, streaming media resource, firewall server resource, ERP server resource, and Java performance monitors.

Step 6: Analyzing test results. During scenario execution, LoadRunner records the performance of the application under different loads. We use LoadRunners graphs and reports to analyze the applications performance.

6. When do you do load and performance Testing?
We perform load testing once we are done with interface (GUI) testing. Modern system architectures are large and complex. Whereas single user testing primarily on functionality and user interface of a system component, application testing focuses on performance and reliability of an entire system. For example, a typical application-testing scenario might depict 1000 users logging in simultaneously to a system. This gives rise to issues such as what is the response time of the system, does it crash, will it go with different software applications and platforms, can it hold so many hundreds and thousands of users, etc. This is when we set do load and performance testing.

7. What are the components of LoadRunner?
The components of LoadRunner are The Virtual User Generator, Controller, and the Agent process, LoadRunner Analysis and Monitoring, LoadRunner Books Online.

8. What Component of LoadRunner would you use to record a Script?
The Virtual User Generator (VuGen) component is used to record a script. It enables you to develop Vuser scripts for a variety of application types and communication protocols.

9. When do you do load and performance Testing?
We perform load testing once we are done with interface (GUI) testing. Modern system architectures are large and complex. Whereas single user testing primarily on functionality and user interface of a system component, application testing focuses on performance and reliability of an entire system. For example, a typical application-testing scenario might depict 1000 users logging in simultaneously to a system. This gives rise to issues such as what is the response time of the system, does it crash, will it go with different software applications and platforms, can it hold so many hundreds and thousands of users, etc. This is when we set do load and performance testing.

10. What are the components of LoadRunner?
The components of LoadRunner are The Virtual User Generator, Controller, and the Agent process, LoadRunner Analysis and Monitoring, LoadRunner Books Online. What Component of LoadRunner would you use to record a Script? - The Virtual User Generator (VuGen) component is used to record a script. It enables you to develop Vuser scripts for a variety of application types and communication protocols.

11. What Component of LoadRunner would you use to play Back the script in multi user mode?
The Controller component is used to playback the script in multi-user mode. This is done during a scenario run where a vuser script is executed by a number of vusers in a group.

12. What is a rendezvous point?
You insert rendezvous points into Vuser scripts to emulate heavy user load on the server. Rendezvous points instruct Vusers to wait during test execution for multiple Vusers to arrive at a certain point, in order that they may simultaneously perform a task. For example, to emulate peak load on the bank server, you can insert a rendezvous point instructing 100 Vusers to deposit cash into their accounts at the same time.

13. What is a scenario?
A scenario defines the events that occur during each testing session. For example, a scenario defines and controls the number of users to emulate, the actions to be performed, and the machines on which the virtual users run their emulations.

14. Explain the recording mode for web Vuser script?
We use VuGen to develop a Vuser script by recording a user performing typical business processes on a client application. VuGen creates the script by recording the activity between the client and the server. For example, in web based applications, VuGen monitors the client end of the database and traces all the requests sent to, and received from, the database server. We use VuGen to: Monitor the communication between the application and the server; Generate the required function calls; and Insert the generated function calls into a Vuser script.

15. Why do you create parameters?
Parameters are like script variables. They are used to vary input to the server and to emulate real users. Different sets of data are sent to the server each time the script is run. Better simulate the usage model for more accurate testing from the Controller; one script can emulate many different users on the system.

16. What is correlation?
Correlation is used to obtain data which are unique for each run of the script and which are generated by nested queries. Correlation provides the value to avoid errors arising out of duplicate values and also optimizing the code (to avoid nested queries). Automatic correlation is where we set some rules for correlation. It can be application server specific. Here values are replaced by data which are created by these rules. In manual correlation, the value we want to correlate is scanned and create correlation is used to correlate.

17. How do you find out where correlation is required?
Two ways: First we can scan for correlations, and see the list of values which can be correlated. From this we can pick a value to be correlated. Secondly, we can record two scripts and compare them. We can look up the difference file to see for the values which needed to be correlated.

18. Where do you set automatic correlation options?
Automatic correlation from web point of view can be set in recording options and correlation tab. Here we can enable correlation for the entire script and choose either issue online messages or offline actions, where we can define rules for that correlation. Automatic correlation for database can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.

19. What is a function to capture dynamic values in the web Vuser script?
Web_reg_save_param function saves dynamic data information to a parameter.

20. Which is VuGen Recording and Scripting language?
LoadRunner script code obtained from recording in the ANSI C language syntax, represented by icons in icon view until you click Script View.

21. What is Scenarios?
Scenarios encapsulate the Vuser Groups and scripts to be executed on load generators at run-time.

Manual scenarios can distribute the total number of Vusers among scripts based on the analyst-specified percentage (evenly among load generators). Goal Oriented scenarios are automatically created based on a specified transaction response time or number of hits/transactions-per-second (TPS). Test analysts specify the % of Target among scripts.

22. When do you disable log in Virtual User Generator, When do you choose standard and extended logs?
Once we debug our script and verify that it is functional, we can enable logging for errors only. When we add a script to a scenario, logging is automatically disabled. Standard Log Option: When you select Standard log, it creates a standard log of functions and messages sent during script execution to use for debugging. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled Extended Log Option: Select extended log to create an extended log, including warnings and other messages. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled. We can specify which additional information should be added to the extended log using the Extended log options.

23. How do you debug a LoadRunner script?
VuGen contains two options to help debug Vuser scripts-the Run Step by Step command and breakpoints. The Debug settings in the Options dialog box allow us to determine the extent of the trace to be performed during scenario execution. The debug information is written to the Output window. We can manually set the message class within your script using the lr_set_debug_message function. This is useful if we want to receive debug information about a small section of the script only.

24. How do you write user defined functions in LR?
Before we create the User Defined functions we need to create the external library (DLL) with the function. We add this library to VuGen bin directory. Once the library is added then we assign user defined function as a parameter. The function should have the following format: __declspec (dllexport) char* (char*, char*)

25. What are the changes you can make in run-time settings?
The Run Time Settings that we make are:
Pacing - It has iteration count.
Log - Under this we have Disable Logging Standard Log and
Extended Think Time - In think time we have two options like Ignore think time and Replay think time.
General - Under general tab we can set the vusers as process or as multithreading and whether each step as a transaction.

26. Where do you set Iteration for Vuser testing?
We set Iterations in the Run Time Settings of the VuGen. The navigation for this is Run time settings, Pacing tab, set number of iterations.

27. How do you perform functional testing under load?
Functionality under load can be tested by running several Vusers concurrently. By increasing the amount of Vusers, we can determine how much load the server can sustain.

28. How to use network drive mappings?
If several load generators need to access the same physical files, rather than having to remember to copy the files each time they change, each load generator can reference a common folder using a mapped drive. But since drive mappings are associated with a specific user:

Logon the load generator as the user the load generator will use
Open Windows Explorer and under Tools select Map a Network Drive and create a drive. It saves time and hassle to have consistent drive letters across load generators, so some organizations reserver certain drive letters for specific locations.
Open the LoadRunner service within Services (accessed from Control Panel, Administrative Tasks),
Click the "Login" tab.
Specify the username and password the load generator service will use. (A dot appears in front of the username if the userid is for the local domain).
Stop and start the service again.

29. What is Ramp up? How do you set this?
This option is used to gradually increase the amount of Vusers/load on the server. An initial value is set and a value to wait between intervals can be specified. To set Ramp Up, go to Scenario Scheduling Options.

30. What is the advantage of running the Vuser as thread?
VuGen provides the facility to use multithreading. This enables more Vusers to be run pergenerator. If the Vuser is run as a process, the same driver program is loaded into memory for each Vuser, thus taking up a large amount of memory. This limits the number of Vusers that can be run on a single generator. If the Vuser is run as a thread, only one instance of the driver program is loaded into memory for the given number of Vusers (say 100). Each thread shares the memory of the parent driver program, thus enabling more Vusers to be run per generator.

31. If you want to stop the execution of your script on error, how do you do that?
The lr_abort function aborts the execution of a Vuser script. It instructs the Vuser to stop executing the Actions section, execute the vuser_end section and end the execution. This function is useful when you need to manually abort a script execution as a result of a specific error condition. When you end a script using this function, the Vuser is assigned the status "Stopped". For this to take effect, we have to first uncheck the Continue on error option in Run-Time Settings.

32. What is the relation between Response Time and Throughput?
The Throughput graph shows the amount of data in bytes that the Vusers received from the server in a second. When we compare this with the transaction response time, we will notice that as throughput decreased, the response time also decreased. Similarly, the peak throughput and highest response time would occur approximately at the same time.

33. Explain the Configuration of your systems?
The configuration of our systems refers to that of the client machines on which we run the Vusers. The configuration of any client machine includes its hardware settings, memory, operating system, software applications, development tools, etc. This system component configuration should match with the overall system configuration that would include the network infrastructure, the web server, the database server, and any other components that go with this larger system so as to achieve the load testing objectives.

34. How do you identify the performance bottlenecks?
Performance Bottlenecks can be detected by using monitors. These monitors might be application server monitors, web server monitors, database server monitors and network monitors. They help in finding out the troubled area in our scenario which causes increased response time. The measurements made are usually performance response time, throughput, hits/sec, network delay graphs, etc.

35. If web server, database and Network are all fine where could be the problem?
The problem could be in the system itself or in the application server or in the code written for the application.

36. How did you find web server related issues?
Using Web resource monitors we can find the performance of web servers. Using these monitors we can analyze throughput on the web server, number of hits per second that occurred during scenario, the number of http responses per second, the number of downloaded pages per second.

37. How did you find database related issues?
By running Database monitor and help of Data Resource Graph we can find database related issues. E.g. you can specify the resource you want to measure on before running the controller and than you can see database related issues.

38. What is the difference between Overlay graph and Correlate graph?
Overlay Graph: It overlay the content of two graphs that shares a common x-axis. Left Y-axis on the merged graph shows the current graphs value & Right Y-axis show the value of Y-axis of the graph that was merged. Correlate Graph: Plot the Y-axis of two graphs against each other. The active graphs Y-axis becomes X-axis of merged graph. Y-axis of the graph that was merged becomes merged graphs Y-axis.

39. How did you plan the Load?
Load test is planned to decide the number of users, what kind of machines we are going to use and from where they are run. It is based on 2 important documents, Task Distribution Diagram and Transaction profile. Task Distribution Diagram gives us the information on number of users for a particular transaction and the time of the load. The peak usage and off-usage are decided from this Diagram. Transaction profile gives us the information about the transactions name and their priority levels with regard to the scenario we are deciding.

40. What does vuser_init action contain?
Vuser_init action contains procedures to login to a server.

41. What does vuser_end action contain?
Vuser_end section contains log off procedures.

42. What is think time?
Think time is the time that a real user waits between actions. Example: When a user receives data from a server, the user may wait several seconds to review the data before responding. This delay is known as the think time. Changing the Threshold: Threshold level is the level below which the recorded think time will be ignored. The default value is five (5) seconds. We can change the think time threshold in the Recording options of the Vugen.

43. What is the difference between standard log and extended log?
The standard log sends a subset of functions and messages sent during script execution to a log. The subset depends on the Vuser type Extended log sends a detailed script execution messages to the output log. This is mainly used during debugging when we want information about:

Parameter substitution
Data returned by the server
Advanced trace

44. What is lr_debug_message?
The lr_debug_message function sends a debug message to the output log when the specified message class is set.

45. What is lr_output_message?
The lr_output_message function sends notifications to the Controller Output window and the Vuser log file.

46. What is lr_error_message?
The lr_error_message function sends an error message to the LoadRunner Output window.

47. What is lrd_stmt?
The lrd_stmt function associates a character string (usually a SQL statement) with a cursor. This function sets a SQL statement to be processed.

48. What is lrd_fetch?
The lrd_fetch function fetches the next row from the result set.

49. What is Throughput?
If the throughput scales upward as time progresses and the number of Vusers increase, this indicates that the bandwidth is sufficient. If the graph were to remain relatively flat as the number of Vusers increased, it would be reasonable to conclude that the bandwidth is constraining the volume of data delivered.

50. What are the various types of Goals in Goal-Oriented Scenario ?
Load Runner provides you with five different types of goals in a goal oriented scenario:

The number of concurrent Vusers

The number of hits per second
The number of transactions per second
The number of pages per minute

The transaction response time that you want your scenario Analysis Scenario (Bottlenecks): In Running Vuser graph correlated with the response time graph you can see that as the number of Vusers increases, the average response time of the check itinerary transaction very gradually increases. In other words, the average response time steadily increases as the load increases. At 56 Vusers, there is a sudden, sharp increase in the average response time. We say that the test broke the server. That is the mean time before failure (MTBF). The response time clearly began to degrade when there were more than 56 Vusers running simultaneously.

Defect Removable Efficiency

2007-11-25T15:16:00.001-08:00

The Defect Removable Efficiency (DRE) is the percentage of defects that have been removed during an activity, computed with the equation below:

DRE = (Number of Defects Removed / Number of Defects at Start of Process) * 100

The DRE can also be computed for each software development activity and plotted on a bar graph to show the relative defect removal efficiencies for each activity. Or, the DRE may be computed for a specific task or technique (e.g. design inspection, code walkthrough, unit test, 6 month operation, etc.)

We can also calculate DRE as:

DRE = A / (A+B)

where A = Defects by raised by testing team and B = Defects raised by customer

If dre <=0.8 then good product otherwise not.

Software Testing Estimation Process

2007-11-25T15:14:00.000-08:00

Software Testing estimation process is one of the most difficult and critical activity. When say that one project will be completed in a particluar time at a particular cost, then it must happen. If it does not happen, it may result in peer's comments and senior management’s warnings to being fired depending on the reasons and seriousness of the failure.

Here are a few rules for effective software testing estimation:

- Estimation must be based on previous projects. All estimation should be based on previous projects.

- Estimation must be recorded. All decisions should be recorded. It is very important because if requirements change for any reason, the records would help the testing team to estimate again.

- Estimation shall be always based on the software requirements. All estimation should be based on what would be tested. The software requirements shall be read and understood by the testing team as well as development team. Without the testing participation, no serious estimation can be considered.

- Estimation must be verified. All estimation should be verified. Two spreadsheets can be created for recording the estimations. At the end, compare both the estimations. If the estimation has any deviation from the recorded ones, then a re-estimation should be made.

- Estimation must be supported by tools. Tools such as spreadsheet containing metrics calculates automatically the costs and duration for each testing phase. Also, a document containing sections such as: cost table, risks, and free notes should be created. Showing this document to customer can help the customer to decide which kind of test he needs.

- Estimation shall be based on expert judgment. The experinced resources can easily make estimate that how long it would take for testing.

- Classifiy the requirements in the following categories:

Critical: The development team has little knowledge in how to implement it.
High: The development team has good knowledge in how to implement it but it is not an easy task.
Normal: The development team has good knowledge in how to implement.

What is the difference between Bug, Error and Defect?

2007-11-25T14:34:00.001-08:00

Error : It is the Deviation from actual and the expected value. Bug : It is found in the development environment before the product is shipped to the respective customer.

Defect : It is found in the product itself after it is shipped to the respective customer.

Explain Peer Review in Software Testing

2007-11-25T14:31:00.000-08:00

It is an alternative form of Testing, where some colleagues were invited to examine your work products for defects and improvement opportunities.

Some Peer review approaches are,

Inspection – It is a more systematic and rigorous type of peer review. Inspections are more effective at finding defects than are informal reviews.
Ex : In Motorola’s Iridium project nearly 80% of the defects were detected through inspections where only 60% of the defects were detected through informal reviews.

Team Reviews – It is a planned and structured approach but less formal and less rigorous comparing to Inspections.

Walkthrough – It is an informal review because the work product’s author describes it to some colleagues and asks for suggestions. Walkthroughs are informal because they typically do not follow a defined procedure, do not specify exit criteria, require no management reporting, and generate no metrics.

Pair Programming – In Pair Programming, two developers work together on the same program at a single workstation and continuously reviewing their work.

Peer Deskcheck – In Peer Deskcheck only one person besides the author examines the work product. It is an informal review, where the reviewer can use defect checklists and some analysis methods to increase the effectiveness.

Passaround – It is a multiple, concurrent peer deskcheck where several people are invited to provide comments on the product.

Seven Tips for Business Rules Project

2007-09-13T15:50:00.000-07:00

Business Rules Management Systems introduce a completely new paradigm into the software systems world in organizations that are new to them.

So, here are 7 tips for Rules project

#1 Identify clearly the Business Decisions you are automating

The first thing that you need to do is to identify, understand and be able to clearly state what the business decisions re, that you are trying to automate. For example, here are some business decision statements!

Decide if an applicant is Eligible for a particular loan
What are the products that I can recommend for a certain applicant
Compute the compensation for a specified employee using his current salary structure
Compute the claim amount that needs to be approved for the given claimant

Doing this upfront clarifies your goals. If this not done, then you run the risk of losing yourself into an Analysis paralysis attack.

#2 Identify your Integration touch points, but worry about it last

I mean it! Integrating the business rules engine is just the easiest thing in the world. For example, with QuickRules.NET or QuickRules java it is about 3 - 10 lines of code depending on the circumstance.

It helps to integrate early because your application shell will be ready.You can fill in the rules later as they are built.

#3 State your rules first. Don’t worry about rule formats or formalism too much to begin with (Rule Capture)

Some of the deadliest fights and confusions happen when people mix up Business rule design and rule requirements.

Your first objective is to determine what are the business policies that form the basis of the business decisions that we stated in #1.

Don’t worry too much about rule formats or formal methods (if you are not comfortable with them). If you can capture them in plain English, Excel tables, visio diagrams, it would just work out fine.

Get this right! You need to understand your rules first. Digest the information, let it sink in, the design will take shape, take my word for it.

If you are not clear about some areas, that is OK too. Remember, your rules are externalized already. You can always add to them later.

#4 Business Rule Design. Keep your Business users in mind when writing the rules.

Once you understand your business policies and how they apply, it is time to start your Business Rule design using formats offered by your BRMS.

Identify and group your rules logically by use case. Keep them modular! Sets of logically related rules are referred to as Rulesets. Identify rulesets first.
Then identify how you would apply the policies in a Ruleset to arrive at the decision that the ruleset applies for. You MUST be able to state it!!
You can capture this rule sequence either using a Flow Rule (sequential & graphical) or as a Rete Ruleset (where the rule execution sequence is determined using the Rete algorithm.
First do Step 2#. You can always change it as you learn more !
Then identify and write the If Then Rules, Decision Tables etc.

NOTE
Business Rules represent your business policies. Your Business owns them. When you write them, keep in mind that the people who need to see them are not techies.

In most organizations, business rule implementation always begins with IT. With experience and maturity, you can hand over rule development, and maintenance over to your business analysts. But that mostly will not happen if this is your first business rules project.

#5 Never forget your data models. Rules finally apply on Business Information

Keep this in mind always. Business Rules Management Systems are also software that work in your application.

If you have stable data models, like say MISMO or your own in-house object model, then go ahead and write your rules directly on your data model.

If your data model is also evolving and you want to allow your rules and data models to evolve independently, then,

Identify your Business terms and capture them using the BRMS. In QuickRules, for example, you can capture business terms as Definitions.
Write your rules on these business terms.
As and when you believe your data model is ready, you can map it to these business terms and your integration is done.
You can keep on remapping them as and when you want to test for a milestone or whatever.

#6 Get business scenario test cases. Get the business to give them to you.

You are writing executable business policies. Remember that! You MUST do this!

Get Test cases for the business decision that you are trying to automate.
If you can, you could even get test cases for each ruleset, which will be best.
You can do one better. Capture the test cases as Excel, and teach your business analysts how to test using their data.
This is an absolute MUST. DO not rely only on application level testing. That is late in the cycle. Test early and upstream, not downstream.

#7 And finally, start small. Don’t over do it.

Not every If Then statement is a Business Rule. A Rule is a Business Rule if and only if it is a policy statement
Remember your reasons for adopting the business rules approach. Visibility for the business, ease of change, centralizing policy management & administration.
Business rules offer a structured approach to capture business policy. Don’t overdo it. To begin with, you may capture only volatile rules using a BRMS
Other business rules that never change, they may be hard wired in the application itself.

### Finally, some things that we forgot to mention above..###

Of course, it is not possible to mention everything in a blog post. Even in a very very long one like this.

I have not covered Policy governance, maintenance, access control, security, versioning issues here. Integration, is something that I have not covered at all.

But the 7 tips I have talked about, they are the basic ones. The others, may be some other day.

The Business Case for Outsourcing to independent software testing company.

2007-08-06T08:14:00.000-07:00

How an independent software testing company can help you improve your quality and
time-to-market while reducing risk and cost

Introduction
Increasingly companies are coming to realize that delivering high-quality software ontime
and on-budget requires that they partner with the best-of-breed vendors for all
aspects of the software development process. These companies want to use more than
one vendor, or multi-source, allowing them to select vendors who are expert in
development, as well as vendors who are expert in software testing.
This paper will explore both the single-source, and the multi-source approach to
application development, highlighting the advantages and disadvantages of each, and
pointing out the distinct advantages that multi-sourcing has over single-sourcing. This
paper will then highlight LogiGear’s strengths, expertise, and thought-leadership as a
preferred multi-source partner for your software testing needs.

Single-Source Application Development
In single source application development, a single vendor is hired to develop an
application or applications, as well as perform all or most of the software testing and
quality assurance (QA) functions. The selection of a single-source vendor may be part of
a larger outsourcing effort or business process outsourcing (BPO) engagement. The goal
in selecting a singe vendor is to achieve the efficiency of only having to deal with a
single vendor, while, of course, driving down costs.

The advantages of having a single source vendor for both software development and
testing are that there is a single point of management, a single service level agreement
(SLA), and a single point of accountability. Many companies incorrectly assume that this
organizational simplicity will further help to drive costs down, while delivering quality
software on-time.

Single source application development is not necessarily the panacea that it would
initially appear to be. There are many trade-offs and potential problems with single
sourcing both application development and software testing. The potential problems with
single sourcing can include:
• Compromising on best-of-breed in testing – When single sourcing, companies
usually focus on selecting a software development partner who also does software
testing. Typically, the primary focus of a vendor is on software development, with
software testing being an additional item for which they can bill. In many
instances, testing is an afterthought, or an adjunct to their primary development
business. It is important to remember that software testing is its own discipline,
with its own methodologies that are separate and distinct from software
development. Treating testing as an afterthought to software development can
potentially lead to quality problems with the resulting software.
• Testing can be in the hands of less-experienced and less-interested engineers -
It is common practice for software development firms to assign their junior staff
to software testing. Testing is often the first step in the career path of new
software engineers who aspire to become software developers. As such, they
typically do not have a testing background, or a good understanding of the
fundamental concepts and practices that make up good software testing, nor do
they want to do testing. Further, they typically do not have, and do not receive,
any formal training in software testing. This can leave one of the most important
parts of the software development process, the testing, in the hands of relative
novices. To make matters worse, these junior testers may be hesitant to point out
problems in the work of their more-experienced co-workers.
• Lack of independence introduces risk into the development process – When
developers and testers are in the same organization, testers can come under
intense pressure to not delay or prevent delivery from occurring. This can be
especially true in an outsourced relationship where meeting or missing delivery
dates may have financial ramifications for the outsourcing vendor. Because of this
dynamic, the customer (outsourcer) may not get true insights into the quality of
the application under development. Even with an SLA, bugs may go unreported
to the customer so that the outsourcing firm can keep its delivery costs down and
meet its delivery commitments.

Multi-Source Application Development
With multi-source application development (multi-sourcing), multiple “best-of-breed”
outsourcing firms are hired, each to perform its own specialized tasks. Using this
approach, a firm would hire both an independent software development company, and an
independent software testing company. Each of these companies would be expert in their
particular discipline.
In such a multi-sourcing arrangement, onshore managers would manage the multiple
vendor relationships and the coordination between the various vendors and themselves. In
many cases, the relationships would be managed at the departmental level with a
company’s development department managing the software development vendor, and the
software testing or quality assurance department managing the testing relationship. Such
a structure would actually create beneficial independence between development and
testing.

There are many advantages in favor of multi-sourcing. These include:
• Companies gain access to best-of-breed firms – By multi-sourcing, a company
can gain access to vendors who are specialized experts in their own discipline, in
this case they would be outsourcing to both a firm expert in development and a
firm expert in software testing.
• Companies gain access to trained testing experts – From the standpoint of
software testing, hiring a dedicated vendor that is strictly focused on software
testing means that a company would be acquiring the services of engineers who
are trained software testing experts. Such engineers understand testing processes,
methodologies, and tools to a high degree of proficiency. This enables them to
perform a more thorough and complete job testing software. In software testing
companies, software testing is not a step towards becoming a developer, but rather
a chosen career path with a clear progression of professional growth.
• Companies lower risk – Risk to the development effort is reduced on many
fronts by multi-sourcing. Companies will be obtaining a high degree of software
testing expertise and professionalism. Companies will also be creating the
necessary and desired independence between the two disciplines, development
and testing, that can contribute to delivering higher quality software. Since the
testing vendor has no “vested interest” in the developed software, they can be
very honest about its quality and readiness for market. Such insight can be
invaluable to company management as they try to weigh quality/time-to-market
trade-offs. Having good information from a testing vendor that you trust can
enable you to make more informed decisions.
• Companies can save money by tapping smaller outsourcing markets –
Breaking up the development process into the components of development and
testing means that a company may be able to tap into smaller outsourcing markets
and vendors, enabling them to lower costs and potentially reducing their exposure
to the engineering turnover (churn) that exists in larger markets and firms.
The main disadvantages of multi-sourcing are that there are multiple vendor
relationships, no single point of contact, and the onshore employees will have to manage
and coordinate the work of multiple companies. This is mitigated by the desired effect of
creating independence between development and testing. Such independence can
inherently contribute to the goal of higher quality software while reducing the risk of
delays, unacceptable post-release support costs, and embarrassing software failures.
These disadvantages can be further minimized or eliminated through the use of effective
testing frameworks that facilitate global teams and good communication. A number of
these tools are commercially available, or they can be developed in-house.

Software Testing hitches and Tips to handle them.

2007-08-01T07:39:00.000-07:00

This is an attempt to identify common software testing hitches, most of the testing projects face and a few tips to overcome them. Software testing is an integral part of the software development life cycle. Inadequately tested applications result in loss of credibility with the customers, be it an existing customer or a new one. It is therefore very essential that effective testing be performed with an intention to eliminate the common problems that might cause havoc before releasing any software.

I. Poor Planning or Estimation
Effective planning is one of the most critical and challenging steps in a test project. Test planning and estimating indicate the order and way in which the tasks are to be carried out and the resources required to execute the same. Proper planning is not possible without a sound and reliable estimate.
1. Effort: Delays can result because of lack of resources to perform the activities in a certain time frame or in less efficient use of resources because too many resources are allocated
2. Schedule: Schedule is estimated after the effort is estimated. Developers underestimate the effort and resources required for testing. As a consequence of which, deadlines are not met or software is delivered only partially tested to the ultimate end user
3. Cost: When budgets are not correctly estimated, it becomes relatively expensive; it might result in some test activities to be cancelled causing more insecurity about the quality of the project

How to tackle?
Take a percentage of the total effort, employ standard ratios in testing based on previous similar test processes, allow for overheads, estimate the hours for individual activities and extrapolate the results. Inadequate testing because of lack of knowledgeable resources such as using testers with little or no experience also results in poor quality of testing. Do not forget to include
1. The training time required to improve the knowledge level of the resources on the domain or technology
2. Buffer time required to resolve any risks that you foresee

II. Ambiguous Requirements
Without adequate documentation, the testing effort usually takes longer and allows more errors to get through to the released version. Ambiguity in requirements makes the test design phase a tedious task. The cost of uncovering and correcting requirement deficiencies at this stage is significantly less than that incurred during the acceptance-testing phase. There may be numerous implied or underlying requirements, which may be overlooked by the testers on glancing the requirements. It is therefore essential that the requirements be understood thoroughly at the initial phase of testing.

How to tackle?
The testers can review the requirements and prepare a list of queries to be addressed on the requirements and get them clarified even before preparing the test cases to enable them deliver a quality product. A report with the deficiencies in requirements may also be prepared.

III. Insufficient Test Coverage
A good test suite will achieve high coverage. Inadequate number of cases cannot complete testing the functionality in its entirety. Test coverage is only a measure of the quality of testing. If high-test coverage is not achieved, it is imperative that the process needs to be strengthened. Another factor to be added is the inadequate test data that does not completely the possible ranges.

How to tackle?
1. The associated test case identification number (say a unique number for every case) can be marked against the requirements in an excel sheet to ensure that test cases are written for all requirements. Low coverage indicates a process problem, which might require test generation technique to be improved, or training to be imparted to the tester. Many tools are available in the market to measure the test coverage.
2. It is not possible to test all conditions in an application system. Data design with valid, invalid data to cover normal processing operations adequately can be prepared. Techniques of boundary values, equivalence partitioning can be applied while preparing test data.

IV. Uncontrolled Test Environment
The more the test environment resembles the final production environment, the more is the reliability of testing. Lack of such an environment will result in unpredictable results while in production.

How to tackle?
1. Testing should take place in a controlled environment. It is therefore separated from the development or production environment. The ownership of test environment should be with the testing team and without their permission no change should happen in the environment.
2. Measures can be taken up to set up the test environment in time and ensure that it is well managed; ie. The test environment should be sufficiently representative for the test to be performed that is, it should be closer to or the same as that of the production environment. It is necessary that the Test Manager or a coordinator manages deliveries from development team and is made responsible for the set up, version management, authorizations etc. If an independent test team is established, it will be ideal to have an independent configuration team also.

V. Testing as an Afterthought
Underestimation of the effort and resources required for testing results in starting testing activities at the fag end of the development cycle when it becomes difficult to fix the major bugs unearthed by the testers and also results in compromising on details in the test documents owing to time constraints.

How to tackle?
Test planning can be initiated as soon as the requirements are defined. Process of test execution in parallel with application development can be adopted.

VI. Inadequate Test documentation
Inadequate/improper test documents (test plans, test specifications, defect reports etc) results in loss of time while analyzing what to be tested/re-tested and the related areas to be tested, which might in turn have an impact in the delivery or the quality of the product.

How to tackle?
1. Adequate effort needs to be spent on documentation also, as test documentation is a very important task in all phases of testing.
2. Care can be taken that all documents related to testing are prepared right from the beginning of the SDLC and updated continuously.

What is actual testing process in practical or company environment?

2007-08-01T07:18:00.000-07:00

Those who get just out of college and start for searching the jobs have this curiosity, How would be the actual working environment in the companies?

Here I focus on software Testing actual working process in the companies. I will try to share more practically rather than theoretically.

Whenever we get any new project there is initial project familiarity meeting. In this meeting we basically discuss on who is client? what is project duration and when is delivery? Who is involved in project i.e manager, Tech leads, QA leads, developers, testers etc etc..?

From the SRS (software requirement specification) project plan is developed. The responsibility of testers is to create software test plan from this SRS and project plan. Developers start coding from the design. The project work is devided into different modules and these project modules are distributed among the developers. In meantime testers responsibility is to create test scenario and write test cases according to assigned modules. We try to cover almost all the functional test cases from SRS. The data can be maintained manually in some excel test case templates or bug tracking tools.

When developers finish individual modules, those modules are assigned to testers. Smoke testing is performed on these modules and if they fail this test, modules are reassigned to respective developers for fix. For passed modules manual testing is carried out from the written test cases. If any bug is found that get assigned to module developer and get logged in bug tracking tool. On bug fix tester do bug verification and regression testing of all related modules. If bug passes the verification it is marked as verified and marked as closed. Otherwise above mentioned bug cycle gets repeated.(I have already cover bug life cycle in other post)

Different tests are performed on individual modules and integration testing on module integration. These tests includes Compatibility testing i.e testing application on different hardware, OS versions, software platform, different browsers etc. Load and stress testing is also carried out according to SRS. Finally system testing is performed by creating virtual client environment. On passing all the test cases test report is prepared and decision is taken to release the product!

So this was a brief outline of process of project life cycle.

Here is detail of each step what testing exactly carried out in each software quality and testing life cycle specified by IEEE and ISO standards:

Review of the software requirement specifications

Objectives is set for the Major releases

Target Date planned for the Releases

Detailed Project Plan is build. This includes the decision on Design Specifications

Develop Test Plan based on Design Specifications

Test Plan : This includes Objectives, Methodology adopted while testing, Features to
be tested and not to be tested, risk criteria, testing schedule, multi-
platform support and the resource allocation for testing.

Test Specifications
This document includes technical details ( Software requirements )
required prior to the testing.

Writing of Test Cases
Smoke(BVT) test cases
Sanity Test cases
Regression Test Cases
Negative Test Cases
Extended Test Cases

Development - Modules developed one by one

Installers Binding: Installers are build around the individual product.

Build procedure :
A build includes Installers of the available products - multiple platforms.

Testing
Smoke Test (BVT) Basic application test to take decision on further testing

Testing of new features
Cross-platform testing
Stress testing and memory leakage testing.

Bug Reporting
Bug report is created

Development - Code freezing
No more new features are added at this point.

Testing
Builds and regression testing.

Decision to release the product
Post-release Scenario for further objectives.

Testing Checklist

2007-08-01T07:11:00.000-07:00

Are you going to start on a new project for testing? Don’t forget to check this Testing Checklist in each and every step of your Project life cycle. List is mostly equivalent to Test plan, it will cover all quality assurance and testing standards.

1 Create System and Acceptance Tests.
2 Start Acceptance test Creation.
3 Identify test team.
4 Create Workplan.
5 Create test Approach.
6 Link Acceptance Criteria and Requirements to form the basis of
acceptance test.
7 Use subset of system test cases to form requirements portion of
acceptance test.
8 Create scripts for use by the customer to demonstrate that the system meets
requirements.
9 Create test schedule. Include people and all other resources.
10 Conduct Acceptance Test.
11 Start System Test Creation.
12 Identify test team members.
13 Create Workplan.
14 Determine resource requirements.
15 Identify productivity tools for testing.
16 Determine data requirements.
17 Reach agreement with data ceter.
18 Create test Approach.
19 Identify any facilities that are needed.
20 Obtain and review existing test material.
21 Create inventory of test items.
22 Identify Design states, conditions, processes, and procedures.
23 Determine the need for Code based (white box) testing. Identify conditions.
24 Identify all functional requirements.
25 End inventory creation.
26 Start test case creation.
27 Create test cases based on inventory of test items.
28 Identify logical groups of business function for new sysyem.
29 Divide test cases into functional groups traced to test item inventory.
30 Design data sets to correspond to test cases.
31 End test case creation.
32 Review business functions, test cases, and data sets with users.
33 Get signoff on test design from Project leader and QA.
34 End Test Design.
35 Begin test Preparation.
36 Obtain test support resources.
37 Outline expected results for each test case.
38 Obtain test data. Validate and trace to test cases.
39 Prepare detailed test scripts for each test case.
40 Prepare & document environmental set up procedures. Include back up and
recovery plans.
41 End Test Preparation phase.
42 Conduct System Test.
43 Execute test scripts.
44 Compare actual result to expected.
45 Document discrepancies and create problem report.
46 Prepare maintenance phase input.
47 Re-execute test group after problem repairs.
48 Create final test report, include known bugs list.
49 Obtain formal signoff.

To Access…. Run Command.

2007-07-23T11:52:00.000-07:00

Accessibility Controls
access.cpl

Add Hardware Wizard
hdwwiz.cpl

Add/Remove Programs
appwiz.cpl

Administrative Tools
control.exe admintools

Automatic Updates
wuaucpl.cpl

Bluetooth Transfer Wizard
fsquirt

Calculator
calc

Certificate Manager
certmgr.msc

Character Map
charmap

Check Disk Utility
chkdsk

Clipboard Viewer
clipbrd

Command Prompt
cmd

Component Services
dcomcnfg

Computer Management
compmgmt.msc

Date and Time Properties
timedate.cpl

DDE Shares
ddeshare

Device Manager
devmgmt.msc

Direct X Control Panel (if installed)*
directx.cpl

Direct X Troubleshooter
dxdiag

Disk Cleanup Utility
cleanmgr

Disk Defragment
dfrg.msc

Disk Management
diskmgmt.msc

Disk Partition Manager
diskpart

Display Properties
control.exe desktop

Display Properties
desk.cpl

Display Properties (w/Appearance Tab Preselected)
control.exe color

Dr. Watson System Troubleshooting Utility
drwtsn32

Driver Verifier Utility
verifier

Event Viewer
eventvwr.msc

File Signature Verification Tool
sigverif

Findfast
findfast.cpl

Folders Properties
control.exe folders

Fonts
control.exe fonts

Fonts Folder
fonts

Free Cell Card Game
freecell

Game Controllers
joy.cpl

Group Policy Editor (XP Prof)
gpedit.msc

Hearts Card Game
mshearts

Iexpress Wizard
iexpress

Indexing Service
ciadv.msc

Internet Properties
inetcpl.cpl

Java Control Panel (if installed)
jpicpl32.cpl

Java Control Panel (if installed)
javaws

Keyboard Properties
control.exe keyboard

Local Security Settings
secpol.msc

Local Users and Groups
lusrmgr.msc

Logs You Out Of W*NDOW$
logoff

Mcft Chat
winchat

Minesweeper Game
winmine

Mouse Properties
control.exe mouse

Mouse Properties
main.cpl

Network Connections
control.exe netconnections

Network Connections
ncpa.cpl

Network Setup Wizard
netsetup.cpl

Nview Desktop Manager (if installed)
nvtuicpl.cpl

Object Packager
packager

ODBC Data Source Administrator
odbccp32.cpl

On Screen Keyboard
osk

Opens AC3 Filter (if installed)
ac3filter.cpl

Password Properties
password.cpl

Performance Monitor
perfmon.msc

Performance Monitor
perfmon

Phone and Modem Options
telephon.cpl

Power Configuration
powercfg.cpl

Printers and Faxes
control.exe printers

Printers Folder
printers

Private Character Editor
eudcedit

Quicktime (If Installed)
QuickTime.cpl

Regional Settings
intl.cpl

Registry Editor
regedit

Registry Editor
regedit32

Removable Storage
ntmsmgr.msc

Removable Storage Operator Requests
ntmsoprq.msc

Resultant Set of Policy
rsop.msc

Resultant Set of Policy (XP Prof)
rsop.msc

Scanners and Cameras
sticpl.cpl

Scheduled Tasks
control.exe schedtasks

Security Center
wscui.cpl

Services
services.msc

Shared Folders
fsmgmt.msc

Shuts Down W*NDOW$
shutdown

Sounds and Audio
mmsys.cpl

Spider Solitare Card Game
spider

SQL Client Configuration
cliconfg

System Configuration Editor
sysedit

System Configuration Utility
msconfig

System File Checker Utility
sfc

System Properties
sysdm.cpl

Task Manager
taskmgr

Telnet Client
telnet

User Account Management
nusrmgr.cpl

Utility Manager
utilman

WINDOWS Firewall
firewall.cpl

WINDOWS Magnifier
magnify

WINDOWS Management Infrastructure
wmimgmt.msc

WINDOWS System Security Tool
syskey

WINDOWS Update Launches
wupdmgr

WINDOWS XP Tour Wizard
tourstart

Wordpad
write

"Testing Relief" 1.0.0 - Test Your Program Code Selectively

2007-07-20T02:48:00.000-07:00

Iron Ant today announces the availability of "Testing Relief" 1.0.0, a unique pretesting tool for applications created on Microsoft .NET Framework. Designed for developers following RUP discipline, "Testing Relief" applies a brand-new approach to software testing, allowing you to increase the quality of software testing while reducing the time needed for it.

One of the challenges in software development is testing an application whose programming code has been modified. It often happens that some parts of the code indirectly affected by the changes remain hidden for the developer, which results in errors. To find a bug takes quite some time, more so if the application has a complex architecture or a large amount of code. In such cases, testing and troubleshooting becomes tiresome and very time-consuming.

"Testing Relief" has been created to address the challenge mentioned above. When set to work, it'll analyze all the changes made to the application, evaluate how the changes influence all parts of the software and single out those parts of the code that depend on the changes but haven't been modified themselves. This will help the developer focus on the "suspicious" parts of the code and find the bug much quicker because, instead of full test of the code, the developer can test the code selectively, only those parts that have been influenced by changes.

This selective approach to testing helps the developer to reduce the time needed for testing the application and increases the quality of testing. It tells you about modifications in the code and influences these modifications have on the rest of the code.

Article sent by Balaji V

Open Source Test Management Tools

2007-07-12T02:00:00.000-07:00

In this post, i will share my views & findings with Open Source Test Management Tools (http://www.opensourcetesting.org/testmgt.php).
Usage of tools in the Test Management is becoming the basic need & It will be tough to manage the activities without tools.

The commercial tools in this segment are costly and may not fit into upcoming organizations budget. So i have been looking at Open Source Test Management tools. In the last week, I have evaluated some open source test management tools.

The following are my requirements for Test Management Tool
Capture the Requirements
Design Test Cases
Map Test Cases with Requirements
Link Bug reports with Test Case ID after the test execution
Test Execution Reports
Version Management for the Test Cases
Search Feature

The following tools are evaluated after the initial screening.
TestLink (http://testlink.sourceforge.net/docs/testLink.php)
RTH (http://www.rth-is-quality.com/home.php)

The above tools work with Apache, MySQL & PHP. Both the tools looks promising and the Test Link has some additional benefits in-terms of better reports and linking with popular issue trackers.

Gulf banks mega-merger

2007-07-12T01:49:00.000-07:00

Emirates Bank International and National Bank of Dubai are merged to create the Gulf's largest bank by assets and market value.
It would have a combined capitalisation of $11.25bn (£5.6bn), topping National Bank of Abu Dhabi - the largest bank by market value in United Arab Emirates.
For more Info:http://news.bbc.co.uk/2/hi/business/6260326.stm

'I'm Testing ...'

2007-07-11T03:00:00.000-07:00

Below are notes on what some of the descriptive words that I use to describe systems do for me. They may do a very different thing for you. And I guess that is to be expected, words are ambiguous.

When I say that I'm Testing [Insert Name of Software Here], that may not mean very much. It may not mean much to other people if they don't know the name, and it certainly doesn't expose the level of knowledge that I have about the system. What it can do for me is get me thinking in terms of the superficialities, of the first things that come to mind:
the adverts,
the basic feature set,
what general kind of things need to be tested
basic requirements

If I extensionalise the name and give it some extra meta-tag information (in this case the version number): I'm testing [Name of Software] version A.B. Then I start to think about:
changes in this version,
what needs to be tested specifically in this version?
what the requirements were for this version?
have all the necessary changes been made?
Delivery dates and timescales.

When I call the Application a 'System' I start to think about
an integration of parts,
interfaces,
data syntax,
flow through the system

When I call the Application 'Software' I start to think of it as a thing:
which runs on an operating system (what versions),
interfaces with hardware (disk),
data semantics
common elements to software (instruction manual, box, help file, install routine).

When I call the Application a 'Solution', I think:
for whom?
for what problem?

And when I say 'Application' I'm thinking:
apply it to what?
Use it for what?
How can I apply it?
What functionality does it have?

IBM agrees to buy software testing firm Telelogic

2007-07-10T05:59:00.000-07:00

IBM has agreed a deal to buy Telelogic in a $745m deal that gives it access to application lifecycle management tools that span the testing, building and governing of software development projects.

Danny Sabbah, IBM Rational general manager, said the move would boost its presence in developing embedded systems for automotive, aerospace and other sectors as those markets demand more complex product designs. Telelogic will operate as part of the Rational developer family, he added.

In a statement, Telelogic CEO Anders Lidbeck said the deal would “provide clients with enhanced capabilities to develop and deploy complex systems on a global basis [through the ability to] leverage a broader set of capabilities".

IBM’s capture of Telelogic will have a huge effect on the software testing world, experts said.

“For both Telelogic and IBM customers [the combination] should ultimately be viewed as offering great potential,” wrote Ovum analyst Bola Rotibi in a research note.

“IBM Rational's testing facilities … have for sometime taken a back seat to the advance facilities offered by Compuware, Mercury and even Borland. Telelogic further strengthens the embedded capabilities [and] provides the boost that IBM Rational guys need to rework their wider testing portfolio. Watch out HP (Mercury), Compuware and Borland.”

Rotibi added that IBM would gain in requirements management where Telelogic’s Doors product is strong, contrasting with IBM’s Rational RequistePro, which has “languished untouched and with no clear strategy or vision of direction”.

However, she cautioned that IBM took a long time to create a roadmap after its acquisition of Rational.

IBM’s purchase of Telelogic also provides it with a greater say in IT governance, the booming sector that aims to help align business and IT processes, for example by visualising integrations and dependencies. Telelogic gained a foothold in IT governance with the acquisition of Popkin Software and its System Architect line in 2005. IBM’s Sabbah said the tools would be “complementary” to its plans.

James Rogers, chief marketing officer of another IT governance firm, Troux Technologies, said, “Popkin was a very strong leader but we have seen Telelgoic moving it towards application lifecycle management. But Doors is the gorilla in requirements management and Telelogic gives IBM access to some vertical markets.”

More broadly, Sabbah noted that IBM “is increasingly relying on software as its engine of growth”, adding that it expects software to generate half of its total profits in three years. The giant has spent about $16bn on over 50 acquisitions over the last five years and Sabbah said that aggressive strategy will continue, including deals that extend “beyond the traditional IT space”.

source www.vnunet.com

Different Types of Testing

2007-07-10T05:35:00.001-07:00

Functionality Testing : The functionality of the application ( i.e GUI components ). against the specifications ( eg, if we click " submit" button, the application should display ..... " confirmation dialog box")

Acceptance testing: Formal testing conducted to determine whether a system satisfies its acceptance criteria and thus whether the customer should accept the system.

Regression Testing: Testing the application for checking whether the previous features are working properly or not, after adding new features to the application.

Stress Testing: The idea of stress testing is to find the breaking point in order to find bugs that will make that break potentially harmful.

Load Testing: This is merely testing at the highest transaction arrival rate in performance testing to see the resource contention, database locks etc...

Black-box Testing: Focuses on the program's functionality against the specification.

White-box Testing: Focuses on the paths of logic.

Unit Testing: The most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses.

Integration Testing - testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.

Incremental Integration Testing: Continuous testing of an application as new functionality is added; requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as needed; done by programmers or by testers.

Functional Testing: Black-box type testing geared to functional requirements of an application; this type of testing should be done by testers.

System Testing: Black-box type testing that is based on overall requirements specifications; covers all combined parts of a system.

Sanity Testing: Typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or destroying databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state.

Performance Testing: This term is often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans.

Usability Testing: Testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers.

Installation/Uninstallation Testing: Testing of full, partial, or upgrade install/uninstall processes.

Security Testing: Testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques.

Compatability Testing: Testing how well software performs in a particular hardware/software/operating system/network/etc. environment.

Ad-hoc Testing: Testing the application in a random manner.

Alpha Testing: Testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.

Beta Testing: Testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.

Definition of Software Error - What is a software error?

2007-07-10T05:31:00.000-07:00

One common definition of a software error is a mismatch between the program and its specification.

Definition #1:
“A mismatch between the program and its specification is an error in the program if and only if the specification exists and is correct.”

Definition #2:
“A software error is present for when the program does not do what its end user reasonability expects to do.” (Myers, 1976)

Categories of Software Errors

User interface errors, such as output errors, incorrect user messages.
Function errors
Defect hardware
Incorrect program version
Testing errors
Requirements errors
Design errors
Documentation errors
Architecture errors
Module interface errors
Performance errors
Error handling
Boundary-related errors
Logic errors such as
calculation errors
State-based behavior errors
Communication errors
Program structure errors, such as control-flow errors

Most programmers are rather cavalier about controlling the quality of the software they write. They bang out some code, run it through some fairly obvious ad hoc tests, and if it seems okay, they’re done. While this approach may work all right for small, personal programs, it doesn’t cut the mustard for professional software development. Modern software engineering practices include considerable effort directed toward software quality assurance and testing. The idea, of course, is to produce completed software systems that have a high probability of satisfying the customer’s needs.

There are two ways to deliver software free of errors. The first is to prevent the introduction of errors in the first place. And the second is to identify the bugs lurking in your code, seek them out, and destroy them. Obviously, the first method is superior. A big part of software quality comes from doing a good job of defining the requirements for the system you’re building and designing a software solution that will satisfy those requirements. Testing concentrates on detecting those errors that creep in despite your best efforts to keep them out.

Defect Severity and Defet Priority

2007-07-10T05:29:00.000-07:00

The defect Severity scale for determining defect criticality and the associated defect Priority levels to be assigned to errors found in software. It is a scale which can be easily adapted to other automated test management tools.

ANSI/IEEE Std 729-1983 Glossary of Software Engineering Terminology defines Criticality as,
"A classification of a software error or fault based on an evaluation of the degree of impact that error or fault on the development or operation of a system (often used to determine whether or when a fault will be corrected)."

The severity framework for assigning defect criticality that has proven most useful in actual testing practice is a five level scale. The criticality associated with each level is based on the answers to several questions.

First, it must be determined if the defect resulted in a system failure. ANSI/IEEE Std 729-1983 defines a failure as,
"The termination of the ability of a functional unit to perform its required function."

Second, the probability of failure recovery must be determined. ANSI/IEEE 729-1983 defines failure recovery as,
"The return of a system to a reliable operating state after failure."

Third, it must be determined if the system can do this on its own or if remedial measures must be implemented in order to return the system to reliable operation.

Fourth, it must be determined if the system can operate reliably with the defect present if it is not manifested as a failure.

Fifth, it must be determined if the defect should or should not be repaired.
The following five level scale of defect criticality addresses the these questions.

The five Levels are:
1. Critical
2. Major
3. Average
4. Minor
5. Exception

1. Critical - The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system.

2. Major - The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system. There is no way to make the failed component(s), however, there are acceptable processing alternatives which will yield the desired result.

3. Average - The defect does not result in a failure, but causes the system to produce incorrect, incomplete, or inconsistent results, or the defect impairs the systems usability.

4. Minor - The defect does not cause a failure, does not impair usability, and the desired processing results are easily obtained by working around the defect.

5. Exception - The defect is the result of non-conformance to a standard, is related to the aesthetics of the system, or is a request for an enhancement. Defects at this level may be deferred or even ignored.
In addition to the defect severity level defined above, defect priority level can be used with severity categories to determine the immediacy of repair.

A five repair priority scale has also be used in common testing practice. The levels are:
1. Resolve Immediately
2. Give High Attention
3. Normal Queue
4. Low Priority
5. Defer

1. Resolve Immediately - Further development and/or testing cannot occur until the defect has been repaired. The system cannot be used until the repair has been effected.

2. Give High Attention - The defect must be resolved as soon as possible because it is impairing development/and or testing activities. System use will be severely affected until the defect is fixed.

3. Normal Queue - The defect should be resolved in the normal course of development activities. It can wait until a new build or version is created.

4. Low Priority - The defect is an irritant which should be repaired but which can be repaired after more serious defect have been fixed.

5. Defer - The defect repair can be put of indefinitely. It can be resolved in a future major system revision or not resolved at all.

Checklist for Test Preparation

2007-07-10T05:25:00.000-07:00

Listed below are questions/suggestions for systematically planning and preparing software testing.

Have you planned for an overall testing schedule and the personnel required, and associated training requirements?

Have the test team members been given assignments?

Have you established test plans and test procedures for
module testing, integration testing, system testing, and
acceptance testing?

Have you designed at least one black-box test case for each system function?

Have you designed test cases for verifying quality objectives/factors (e.g. reliability, maintainability, etc.)?

Have you designed test cases for verifying resource objectives?

Have you defined test cases for performance tests, boundary tests, and usability tests?

Have you designed test cases for stress tests (intentional attempts to break system)?

Have you designed test cases with special input values (e.g. empty files)?

Have you designed test cases with default input values?

Have you described how traceability of testing to requirements is to be demonstrated (e.g. references to the specified functions and requirements)?

Do all test cases agree with the specification of the function or requirement to be tested?

Have you sufficiently considered error cases? Have you designed test cases for invalid and unexpected input conditions as well as valid conditions?

Have you defined test cases for white-box-testing (structural tests)?

Have you stated the level of coverage to be achieved by structural tests?

Have you unambiguously provided test input data and expected test results or expected messages for each test case?

Have you documented the purpose of and the capability demonstrated by each test case?

Is it possible to meet and to measure all test objectives defined (e.g. test coverage)?

Have you defined the test environment and tools needed for executing the software test?

Have you described the hardware configuration an resources needed to implement the designed test cases?

Have you described the software configuration needed to implement the designed test cases?

Have you described the way in which tests are to be recorded?

Have you defined criteria for evaluating the test results?

Have you determined the criteria on which the completion of the test will be judged?

Have you considered requirements for regression testing?

Developing a Test Specification

2007-07-10T04:45:00.000-07:00

Definitions
I’ve seen the terms “Test Plan” and “Test Specification” mean slightly different things over the years. In a formal sense (at this given point in time for me), we can define the terms as follows:

Test Specification – a detailed summary of what scenarios will be tested, how they will be tested, how often they will be tested, and so on and so forth, for a given feature. Examples of a given feature include, “Intellisense, Code Snippets, Tool Window Docking, IDE Navigator.” Trying to include all Editor Features or all Window Management Features into one Test Specification would make it too large to effectively read.
Test Plan – a collection of all test specifications for a given area. The Test Plan contains a high-level overview of what is tested (and what is tested by others) for the given feature area. For example, I might want to see how Tool Window Docking is being tested. I can glance at the Window Management Test Plan for an overview of how Tool Window Docking is tested, and if I want more info, I can view that particular test specification.
If you ask a tester on another team what’s the difference between the two, you might receive different answers. In addition, I use the terms interchangeably all the time at work, so if you see me using the term “Test Plan”, think “Test Specification.”

Parts of a Test Specification
A Test Specification should consist of the following parts:
History / Revision - Who created the test spec? Who were the developers and Program Managers (Usability Engineers, Documentation Writers, etc) at the time when the test spec was created? When was it created? When was the last time it was updated? What were the major changes at the time of the last update?

Feature Description – a brief description of what area is being tested.
What is tested? – a quick overview of what scenarios are tested, so people looking through this specification know that they are at the correct place.

What is not tested? - are there any areas being covered by different people or different test specs? If so, include a pointer to these test specs.

Nightly Test Cases – a list of the test cases and high-level description of what is tested each night (or whenever a new build becomes available). This bullet merits its own blog entry. I’ll link to it here once it is written.

Breakout of Major Test Areas - This section is the most interesting part of the test spec where testers arrange test cases according to what they are testing. Note: in no way do I claim this to be a complete list of all possible Major Test Areas. These areas are examples to get you going.

Specific Functionality Tests – Tests to verify the feature is working according to the design specification. This area also includes verifying error conditions.

Security tests – any tests that are related to security. An excellent source for populating this area comes from the Writing Secure Code book.

Accessibility Tests – This section shouldn’t be a surprised to any of my blog readers. See The Fundamentals of Accessibility for more info.

Stress Tests – This section talks about what tests you would apply to stress the feature.

Performance Tests - this section includes verifying any perf requirements for your feature.

Edge cases – This is something I do specifically for my feature areas. I like walking through books like How to break software, looking for ideas to better test my features. I jot those ideas down under this section

Localization / Globalization - tests to ensure you’re meeting your product’s International requirements.

Setting Test Case Priority

A Test Specification may have a couple of hundred test cases, depending on how the test cases were defined, how large the feature area is, and so forth. It is important to be able to query for the most important test cases (nightly), the next most important test cases (weekly), the next most important test cases (full test pass), and so forth. A sample prioritization for test cases may look like:

Highest priority (Nightly) – Must run whenever a new build is available

Second highest priority (Weekly) – Other major functionality tests run once every three or four builds

Lower priority – Run once every major coding milestone

What is User Acceptance Testing?

2007-07-10T04:39:00.000-07:00

Concept
UAT goes under many names. As well as user acceptance testing, it is also known as Beta Testing (usually in the PC world) QA Testing, Application Testing, End User Testing.

Developing software is a expensive business. It is expensive in ;
time, as the software must be analysed, specified, designed and written
people, as very few development projects are one man jobs
money, the people responsible for the analysis, specification and development of software do not come cheap (look at the current rates for contractors!)
If having expended all these peoples time and the company's money, if the resulting software is not completely suitable to the purpose required, then that time and money has not been fully utilised.

If the software is suitable to the purpose, but; does not dovetail precisely with the business processes makes processes more difficult to do than before causes business processes to take longer than previously
makes additional processes necessary, with making other processes obsolete then you may not will not see return on your investment in the software until much later, or may not even see a return on your investment.

Question : how do we ensure that we do not end up in this situation?
Answer : we test the software against objective criteria to ensure that we don't
Previously, most testing was left in the hands of the development teams, with the end users trusting those teams to deliver applications that were not only fully functional and stable, but also applications that would dovetail into business processes, and support those processes (maybe even make things a little bit easier)

However, the testing executed by developers is to ensure that the code they have created is stable and functional. They will test that;

they cover all the lines and logic paths through their code
all the screens flow backwards and forwards in the correct order
the software meets the functions specified (eg calculations are correct, reports have correct columns, screens have correct validation etc)
This testing might not be done through the application itself (often because it has not been completely built while they are testing), so they will only add a few records, maybe by editing the file/table and adding the records, rather than using the 'Record Entry Screen'.

THIS IS NOT A PROBLEM.
I AM NOT DEGRADING THE SYSTEM AND UNIT TESTING DONE BY THE DEVELOPERS
As we will see later, this does not pose us a problem, because the UAT department will cover this testing. This system testing and unit testing by the developers is still very valid and useful. I would rather take delivery of an application where the development team say "We have done this, done that, hacked this file and added a few records, ran a few test cases through here - and everything seems OK", then take an application which has not gone through any system testing.

The application that has been tested by the developers will have had most of the obvious flaws identified and ironed out, and only the types of issues the testing was designed for should be identified. The second application will be completely unknown, and some of the time allocated for UAT will be spent identifying and fixing problems problems that could have been easily identified and rectified by the developers.

Also, because the developers are testing their own work, there is a tendency for them to skip areas because they 'know that there won't be a problem there'.

I have spoke to developers who have came to our company from places that do not do UAT, and they are both impressed with how we do things, but also like the idea of an independent third party testing their software.

These people are professional software developers, and they do not want to supply something that isn't exactly what's wanted, and they feel that the UA testing gives them a large comfort zone that any problems with their work will be identified and escalated back to them for correction.

As I said, these issues do not prove a problem to the user acceptance tester.

The four issues of the software delivered not matching the business process, making things more difficult etc are circumvented by the user acceptance tester.

While the developer tests against the system specification and technical documentation, the user acceptance tester test against the business requirements. The former tests the code, the latter the application. We will come to the test planning in a bit.

The issue of the developer testing their own work ceases to be an issue, as the UAT team will design a testing strategy that covers all areas of the business requirements, whether or not the developer feels there may be problems in a specific area.

The issue of additional processes being necessary should also not be a problem. As I said before, the UAT team tests the application against the business requirements, so all testing is done through the use of the proper system transactions.

The UAT team do not hack tables / file to create data, if a client record is needed for a test, then the UAT will create this client, by use of the formal client maintenance transaction, not by adding a record to the 'Client_Details' file.

This use of the formal application transaction transactions serves two purposes

it tests all the transactions that the business users shall run, giving complete 'business' coverage (as opposed to code coverage, or logic path coverage)
it will highlight any potential areas of adverse impacts on the business processes. If the contents of a form (eg an application form for a new life assurance policy) is used as the basis for creating a new life assurance policy record, then the use of the formal 'New Life Assurance Policy' transaction will determine whether the transaction works, and also whether the form holds the requisite information to create the policy records.
The 'New Life Assurance Policy' system may require the client to declare whether they smoke or not, however, if this question is not on the application form, then the business users will have to re-contact every new client, to determine whether or not they smoke!

We can see then, that it is the role of user acceptance testing to not only prove whether or not an application works, but also to prove how it will fit with business processes.

User Acceptance Testing Processes
OK, now we have determined what UAT is, now we need to look at HOW we achieve these objectives.

The user acceptance test life cycle follows the path shown below (obviously at a very high level);

analysis of business requirements We can't do anything concerning testing until we understand what the developments are supposed to achieve. This is quite an intangible step in the process, and consists mostly of thought processes, meeting etc. The end result, is a clear vision,in the testers mind, of what they are going to be expected to prove, and why it is necessary.
analysis of testing requirements. This is more tangible than the first stage, and consists of documenting the areas of the development that require testing, the methodologies you will need to use to test them, and the results to expect to be returned when you test them.
Execution of testing. Doing the business. This is what it all boils down to. Every development project will be different, and you will have had enough experience in this part of the cycle to not need any pointers from me!
Getting the testing signed off. There is no use going through all of these processes, raising problems to developments teams, having more work done by the development teams in fixing those problems, re-testing the changes and re-doing all your regression scripts, unless at the end of the day, you can the users to sign off the changes.

What is a Test Case?

2007-07-10T04:33:00.000-07:00

Definition of Test Case

- In software engineering, a test case is a set of conditions or variables under which a tester will determine if a requirement upon an application is partially or fully satisfied. It may take many test cases to determine that a requirement is fully satisfied. In order to fully test that all the requirements of an application are met, there must be at least one test case for each requirement unless a requirement has sub requirements. In that situation, each sub requirement must have at least one test case.

More Definitions of a Test Case.

- A test case is also defined as a sequence of steps to test the correct behavior of a functionality/feature of an application.

- A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.

- A test case is a list of the conditions or issues of what the tester want to test in a software. Test case helps to come up with test data. A test case has an input description, Test sequence and an expected behavior.

The characteristics of a test case is that there is a known input and an expected output, which is worked out before the test. The known input should test a pre-condition and the expected output should test a post-condition.

Under special circumstances, there could be a need to run the test, produce results - and a team of experts evaluate if the results can be considered as passed. The first test is taken as the base line for subsequent test / product release cycles.

Test cases include a description of the functionality to be tested taken from either the requirements or use cases, and the preparation required to ensure that the test can be conducted.

Roles in software testing

2007-07-07T08:57:00.000-07:00

Software testing can be done by software testers. Until the 1950s the term software tester was used generally, but later it was also seen as a separate profession. Regarding the periods and the different goals in software testing (see D. Gelperin and W.C. Hetzel) there have been established different roles: test lead/manager, tester, test designer, test automater/automation developer, test administrator. participants of testing team
1.Tester
2.Developer
3.Customer
4.Information Service Management
5.Senior Organization Management
6.Quality team

Certification

2007-07-07T08:50:00.000-07:00

Several certification programs exist to support the professional aspirations of software testers and quality assurance specialists. No certification currently offered actually requires the applicant to demonstrate the ability to test software. No certification is based on a widely accepted body of knowledge. No certification board decertifies individuals.^{[verification needed]}^{[citation needed]} This has led some to declare that the testing field is not ready for certification . Certification itself cannot measure an individual's productivity, their skill, or practical knowledge, and cannot guarantee their competence, or professionalism as a tester .

Open Certification of Software Testers

Certifications can be grouped into: exam-based and education-based. Exam-based certifications: For these there is the need to pass an exam, which can also be learned by self-study: e.g. for ISTQB or QAI. Education-based certifications are instructor-led sessions, where each course has to be passed, e.g. IIST (International Institute for Software Testing).

Testing certifications

CSQE offered by the American Society for Quality
CSTE offered by the Quality Assurance Institute (QAI)
CSTP offered by the International Institute for Software Testing
ISEB offered by the Information Systems Examinations Board
ISTQB offered by the International Software Testing Qualification Board

Quality assurance certifications

CSQA offered by the Quality Assurance Institute (QAI)

Manual vs. Automated

2007-07-07T08:49:00.001-07:00

Some writers believe that test automation is so expensive relative to its value that it should be used sparingly. Others recommend automating 100% of all tests. A challenge with automation is that automated testing requires automated test oracles (an oracle is a mechanism or principle by which a problem in the software can be recognized). Such tools have value in load testing software (by signing on to an application with hundreds or thousands of instances simultaneously), or in checking for intermittent errors in software. The success of automated software testing depends on complete and comprehensive test planning. Software development strategies such as test-driven development are highly compatible with the idea of devoting a large part of an organization's testing resources to automated testing. Many large software organizations perform automated testing. Some have developed their own automated testing environments specifically for internal development, and not for resale.

Exploratory vs. Scripted

2007-07-07T08:48:00.000-07:00

Exploratory testing means simultaneous, test design, and test execution, with an emphasis on learning. Scripted testing means that learning and test design happens prior to test execution, and quite often the learning has to be done again during test execution. Exploratory testing is very common, but in most writing and training about testing it is barely mentioned and generally misunderstood. Some writers consider it a primary and essential practice. Structured exploratory testing is a compromise when the testers are familiar with the software. A vague test plan, known as a test charter, is written up, describing what functionalities need to be tested but not how, allowing the individual testers to choose the method and steps of testing.

There are two main disadvantages associated with a primarily exploratory testing approach. The first is that there is no opportunity to prevent defects, which can happen when the designing of tests in advance serves as a form of structured static testing that often reveals problems in system requirements and design. The second is that, even with test charters, demonstrating test coverage and achieving repeatability of tests using a purely exploratory testing approach is difficult. For this reason, a blended approach of scripted and exploratory testing is often used to reap the benefits of while mitigating each approach's disadvantages.

Agile vs. Traditional

2007-07-07T08:47:00.002-07:00

Starting around 1990, a new style of writing about testing began to challenge what had come before. The seminal work in this regard is widely considered to be Testing Computer Software, by Cem Kaner (1988, ISBN 0-8306-9563-X; as of 1999 in a 3rd edition, ISBN 1-85032-908-7). Instead of assuming that testers have full access to source code and complete specifications, these writers, including Kaner and James Bach, argued that testers must learn to work under conditions of uncertainty and constant change. Meanwhile, an opposing trend toward process "maturity" also gained ground, in the form of the Capability Maturity Model. The agile testing movement (which includes but is not limited to forms of testing practiced on agile development

projects) has popularity mainly in commercial circles, whereas the CMM was embraced by government and military software providers.

However, saying that "maturity models" like CMM gained ground against or opposing Agile testing may not be right. Agile movement is a 'way of working', while CMM are a process improvement idea.

There is however another point of view that must be considered. That is the operational culture of an organization. While it may be true that testers must have an ability to work in a world of uncertainty, it is also true that their flexibility must have direction. In many cases test cultures are self-directed and as a result fruitless, unproductive results can ensue. Furthermore, providing of positive defect evidence may only reflect either the tip of a much larger problem or that you have exhausted all possibilities. A framework is a test of Testing. It provides a boundary that can measure (validate) the capacity of our work. Both sides have, and will continue to argue the virtues of their work. The proof however is in each and every assessment of delivery quality. It does little good to test systematically if you are too narrowly focused. On the other hand, finding a bunch of errors is not an indicator that Agile methods was the driving force; you may simply have stumbled upon an obviously poor piece of work.

Controversy

2007-07-07T08:47:00.001-07:00

There is considerable controversy among testing writers and consultants about what constitutes responsible software testing. Members of the "context-driven" school of testing believe that there are no "best practices" of testing, but rather that testing is a set of skills that allow the tester to select or invent testing practices to suit each unique situation. This belief directly contradicts standards such as the IEEE 829 test documentation standard, and organizations such as the Food and Drug Administration who promote them.

Code Coverage

2007-07-07T08:45:00.001-07:00

Code coverage is inherently a white box testing activity. The target software is built with special options or libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) are mapped back to the function points in the source code. This process allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested.

Test engineers can look at code coverage test results to help them devise test cases and input or configuration sets that will increase the code coverage over vital functions. Two common forms of code coverage used by testers are statement (or line) coverage, and path (or edge) coverage. Line coverage reports on the execution footprint of testing in terms of which lines of code were executed to complete the test. Edge coverage reports which branches, or code decision points were executed to complete the test. They both report a coverage metric, measured as a percentage.

Generally code coverage tools and libraries exact a performance and/or memory or other resource cost which is unacceptable to normal operations of the software. Thus they are only used in the lab. As one might expect there are classes of software that cannot be feasibly subjected to these coverage tests, though a degree of coverage mapping can be approximated through analysis rather than direct testing.

There are also some sorts of defects which are affected by such tools. In particular some race conditions or similar real time sensitive operations can be masked when run under code coverage environments; and conversely some of these defects may become easier to find as a result of the additional overhead of the testing code.

A sample testing cycle

2007-07-07T08:44:00.002-07:00

Although testing varies between organizations, there is a cycle to testing:

Requirements Analysis: Testing should begin in the requirements phase of the software development life cycle.
During the design phase, testers work with developers in determining what aspects of a design are testable and under what parameter those tests work.
Test Planning: Test Strategy, Test Plan(s), Test Bed creation.
Test Development: Test Procedures, Test Scenarios, Test Cases, Test Scripts to use in testing software.
Test Execution: Testers execute the software based on the plans and tests and report any errors found to the development team.
Test Reporting: Once testing is completed, testers generate metrics and make final reports on their test effort and whether or not the software tested is ready for release.
Retesting the Defects

Not all errors or defects reported must be fixed by a software development team. Some may be caused by errors in configuring the test software to match the development or production environment. Some defects can be handled by a workaround in the production environment. Others might be deferred to future releases of the software, or the deficiency might be accepted by the business user. There are yet other defects that may be rejected by the development team (of course, with due reason) if they deem it inappropriate to be called a defect.

Test cases, suites, scripts and scenarios

2007-07-07T08:44:00.001-07:00

A test case is a software testing document,which consists of event, action, input, output, expected result and actual result. Clinically defined (IEEE 829-1998) a test case is an input and an expected result. This can be as pragmatic as 'for condition x your derived result is y', whereas other test cases described in more detail the input scenario and what results might be expected. It can occasionally be a series of steps (but often steps are contained in a separate test procedure that can be exercised against multiple test cases, as a matter of economy) but with one expected result or expected outcome. The optional fields are a test case ID, test step or order of execution number, related requirement(s), depth, test category, author, and check boxes for whether the test is automatable and has been automated. Larger test cases may also contain prerequisite states or steps, and descriptions. A test case should also contain a place for the actual result. These steps can be stored in a word processor document, spreadsheet, database or other common repository. In a database system, you may also be able to see past test results and who generated the results and the system configuration used to generate those results. These past results would usually be stored in a separate table.

The term test script is the combination of a test case, test procedure and test data. Initially the term was derived from the byproduct of work created by automated regression test tools. Today, test scripts can be manual, automated or a combination of both.

The most common term for a collection of test cases is a test suite. The test suite often also contains more detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester identifies the system configuration used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests.

Collections of test cases are sometimes incorrectly termed a test plan. They might correctly be called a test specification. If sequence is specified, it can be called a test script, scenario or procedure.

Test levels

2007-07-07T08:43:00.001-07:00

Unit testing tests the minimal software component, or module.
Integration testing exposes defects in the interfaces and interaction between integrated components (modules).
Functional testing tests the product according to programmable work.
System testing tests an integrated system to verify/validate that it meets its requirements.
Acceptance testing can be conducted by the client. It allows the end-user, customer or client to decide whether or not to accept the product. Acceptance testing may be performed after the testing and before the implementation phase.
- Alpha testing is simulated or actual operational testing by potential users/customers or an independent test team at the developers' site. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing, before the software goes to beta testing.
- Beta testing comes after alpha testing. Versions of the software, known as beta versions, are released to a limited audience outside of the company. The software is released to groups of people so that further testing can ensure the product has few faults or bugs. Sometimes, beta versions are made available to the open public to increase the feedback field to a maximal number of future users.

It should be noted that although both Alpha and Beta are referred to as testing it is in fact use immersion. The rigors that are applied are often unsystematic and many of the basic tenets of testing process are not used. The Alpha and Beta period provides insight into environmental and utilization conditions that can impact the software.

After modifying software, either for a change in functionality or to fix defects, a regression test re-runs previously passing tests on the modified software to ensure that the modifications haven't unintentionally caused a regression of previous functionality. Regression testing can be performed at any or all of the above test levels. These regression tests are often automated.

White-box and black-box testing

2007-07-07T08:42:00.000-07:00

White box and black box testing are terms used to describe the point of view a test engineer takes when designing test cases. Black box being an external view of the test object and white box being an internal view. Software testing is partly intuitive, but largely systematic. Good testing involves much more than just running the program a few times to see whether it works. Thorough analysis of the program under test, backed by a broad knowledge of testing techniques and tools are prerequisites to systematic testing. Software Testing is the process of executing software in a controlled manner; in order to answer the question “Does this software behave as specified?” Software testing is used in association with verification and validation. Verification is the checking of or testing of items, including software, for conformance and consistency with an associated specification. Software testing is just one kind of verification, which also uses techniques such as reviews, inspections, and walk-throughs. Validation is the process of checking what has been specified is what the user actually wanted.

Validation: Are we doing the right job?
Verification: Are we doing the job right?

In order to achieve consistency in testing, it is imperative to have and follow a set of testing principles. This enhances the efficiency of testing within SQA team members and thus contributes to increased productivity.

There are generally 3 main levels of software testing carried out:

Unit Testing: in which each unit (basic component) of the software is tested to verify that the detailed design for the unit has been correctly implemented
Integration testing: in which progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a whole.
System testing: in which the software is integrated to the overall product and tested to show that all requirements are met

A further level of testing is also done, in accordance with requirements:

Acceptance testing: upon which the acceptance of the complete software is based. The clients often do this.

Changes made in order to fix bugs or develop further fucntionality may cause previously successful tests to fail, so regression testing is used to repeat the earlier successful tests to ensure that changes made in the software have not introduced new bugs/side effects.

In recent years the term grey box testing has come into common usage. The typical grey box tester is permitted to set up or manipulate the testing environment, like seeding a database, and can view the state of the product after his actions, like performing a SQL query on the database to be certain of the values of columns. It is used almost exclusively of client-server testers or others who use a database as a repository of information, but can also apply to a tester who has to manipulate XML files (DTD or an actual XML file) or configuration files directly, or perform testing like SQL injection. It can also be used by testers who know the internal workings or algorithm of the software under test and can write tests specifically for the anticipated results. For example, testing a data warehouse implementation involves loading the target database with information, and verifying the correctness of data population and loading of data into the correct tables.

History of TESTING

2007-07-07T08:40:00.000-07:00

The separation of debugging from testing was initially introduced by Glenford J. Myers in his 1978 book the "Art of Software Testing". Although his attention was on breakage testing it illustrated the desire of the software engineering community to separate fundamental development activities, such as debugging, from that of verification. Drs. Dave Gelperin and William C. Hetzel classified in 1988 the phases and goals in software testing as follows: until 1956 it was the debugging oriented period, where testing was often associated to debugging: there was no clear difference between testing and debugging. From 1957-1978 there was the demonstration oriented period where debugging and testing was distinguished now - in this period it was shown, that software satisfies the requirements. The time between 1979-1982 is announced as the destruction oriented period, where the goal was to find errors. 1983-1987 is classified as the evaluation oriented period: intention here is that during the software lifecycle a product evaluation is provided and measuring quality. From 1988 on it was seen as prevention oriented period where tests were to demonstrate that software satisfies its specification, to detect faults and to prevent faults. Dr. Gelperin chaired the IEEE 829-1989 (Test Documentation Standard) with Dr. Hetzel writing the book "The Complete Guide of Software Testing". Both works were pivotal in to today's testing culture and remain a consistent source of reference. Dr. Gelperin and Jerry E. Durant also went on to develop High Impact Inspection Technology that builds upon traditional Inspections but utilizes a test driven additive.

How can World Wide Web sites be tested?

2007-07-07T08:37:00.001-07:00

Web sites are essentially client/server applications - with web servers and 'browser' clients. Consideration should be given to the interactions between html pages, TCP/IP communications, Internet connections, firewalls, applications that run in web pages (such as applets, javascript, plug-in applications), and applications that run on the server side (such as cgi scripts, database interfaces, logging applications, dynamic page generators, asp, etc.). Additionally, there are a wide variety of servers and browsers, various versions of each, small but sometimes significant differences between them, variations in connection speeds, rapidly changing technologies, and multiple standards and protocols. The end result is that testing for web sites can become a major ongoing effort. Other considerations might include:

What are the expected loads on the server (e.g., number of hits per unit time?), and what kind of performance is required under such loads (such as web server response time, database query response times). What kinds of tools will be needed for performance testing (such as web load testing tools, other tools already in house that can be adapted, web robot downloading tools, etc.)?
Who is the target audience? What kind of browsers will they be using? What kind of connection speeds will they by using? Are they intra- organization (thus with likely high connection speeds and similar browsers) or Internet-wide (thus with a wide variety of connection speeds and browser types)?
What kind of performance is expected on the client side (e.g., how fast should pages appear, how fast should animations, applets, etc. load and run)?
Will down time for server and content maintenance/upgrades be allowed? how much?
What kinds of security (firewalls, encryptions, passwords, etc.) will be required and what is it expected to do? How can it be tested?
How reliable are the site's Internet connections required to be? And how does that affect backup system or redundant connection requirements and testing?
What processes will be required to manage updates to the web site's content, and what are the requirements for maintaining, tracking, and controlling page content, graphics, links, etc.?
Which HTML specification will be adhered to? How strictly? What variations will be allowed for targeted browsers?
Will there be any standards or requirements for page appearance and/or graphics throughout a site or parts of a site??
How will internal and external links be validated and updated? how often?
Can testing be done on the production system, or will a separate test system be required? How are browser caching, variations in browser option settings, dial-up connection variabilities, and real-world internet 'traffic congestion' problems to be accounted for in testing?
How extensive or customized are the server logging and reporting requirements; are they considered an integral part of the system and do they require testing?
How are cgi programs, applets, javascripts, ActiveX components, etc. to be maintained, tracked, controlled, and tested?

Some sources of site security information include the Usenet newsgroup 'comp.security.announce' and links concerning web site security

Some usability guidelines to consider - these are subjective and may or may not apply to a given situation

Pages should be 3-5 screens max unless content is tightly focused on a single topic. If larger, provide internal links within the page.
The page layouts and design elements should be consistent throughout a site, so that it's clear to the user that they're still within a site.
Pages should be as browser-independent as possible, or pages should be provided or generated based on the browser-type.
All pages should have links external to the page; there should be no dead-end pages.
The page owner, revision date, and a link to a contact person or organization should be included on each page.

What if there isn't enough time for thorough testing?

2007-07-07T08:36:00.001-07:00

Use risk analysis to determine where testing should be focused.
Since it's rarely possible to test every possible aspect of an application, every possible combination of events, every dependency, or everything that could go wrong, risk analysis is appropriate to most software development projects. This requires judgement skills, common sense, and experience. (If warranted, formal methods are also available.) Considerations can include:

Which functionality is most important to the project's intended purpose?
Which functionality is most visible to the user?
Which functionality has the largest safety impact?
Which functionality has the largest financial impact on users?
Which aspects of the application are most important to the customer?
Which aspects of the application can be tested early in the development cycle?
Which parts of the code are most complex, and thus most subject to errors?
Which parts of the application were developed in rush or panic mode?
Which aspects of similar/related previous projects caused problems?
Which aspects of similar/related previous projects had large maintenance expenses?
Which parts of the requirements and design are unclear or poorly thought out?
What do the developers think are the highest-risk aspects of the application?
What kinds of problems would cause the worst publicity?
What kinds of problems would cause the most customer service complaints?
What kinds of tests could easily cover multiple functionalities?
Which tests will have the best high-risk-coverage to time-required ratio?

How can it be known when to stop testing?

2007-07-07T08:35:00.002-07:00

This can be difficult to determine. Many modern software applications are so complex, and run in such an interdependent environment, that complete testing can never be done. Common factors in deciding when to stop are:

Deadlines (release deadlines, testing deadlines, etc.)
Test cases completed with certain percentage passed
Test budget depleted
Coverage of code/functionality/requirements reaches a specified point
Bug rate falls below a certain level
Beta or alpha testing period ends

What can be done if requirements are changing continuously?

2007-07-07T08:35:00.001-07:00

This is a common problem for organizations where there are expectations that requirements can be pre-determined and remain stable. If these expectations are reasonable, here are some approaches:

Work with the project's stakeholders early on to understand how requirements might change so that alternate test plans and strategies can be worked out in advance, if possible.
It's helpful if the application's initial design allows for some adaptability so that later changes do not require redoing the application from scratch.
If the code is well-commented and well-documented this makes changes easier for the developers.
Use some type of rapid prototyping whenever possible to help customers feel sure of their requirements and minimize changes.
The project's initial schedule should allow for some extra time commensurate with the possibility of changes.
Try to move new requirements to a 'Phase 2' version of an application, while using the original requirements for the 'Phase 1' version.
Negotiate to allow only easily-implemented new requirements into the project, while moving more difficult new requirements into future versions of the application.
Be sure that customers and management understand the scheduling impacts, inherent risks, and costs of significant requirements changes. Then let management or the customers (not the developers or testers) decide if the changes are warranted - after all, that's their job.
Balance the effort put into setting up automated testing with the expected effort required to refactor them to deal with changes.
Try to design some flexibility into automated test scripts.
Focus initial automated testing on application aspects that are most likely to remain unchanged.
Devote appropriate effort to risk analysis of changes to minimize regression testing needs.
Design some flexibility into test cases (this is not easily done; the best bet might be to minimize the detail in the test cases, or set up only higher-level generic-type test plans)
Focus less on detailed test plans and test cases and more on ad hoc testing (with an understanding of the added risk that this entails).

If this is a continuing problem, and the expectation that requirements can be pre-determined and remain stable is NOT reasonable, it may be a good idea to figure out why the expectations are not aligned with reality, and to refactor an organization's or project's software development process to take this into account. It may be appropriate to consider agile development approaches.

What if the application has functionality that wasn't in the requirements?

2007-07-07T08:34:00.001-07:00

It may take serious effort to determine if an application has significant unexpected or hidden functionality, and it could indicate deeper problems in the software development process. If the functionality isn't necessary to the purpose of the application, it should be removed, as it may have unknown impacts or dependencies that were not taken into account by the designer or the customer. (If the functionality is minor and low risk then no action may be necessary.) If not removed, information will be needed to determine risks and to determine any added testing needs or regression testing needs. Management should be made aware of any significant added risks as a result of the unexpected functionality.

This problem is a standard aspect of projects that include COTS (Commercial Off-The-Shelf) software or modified COTS software. The COTS part of the project will typically have a large amount of functionality that is not included in project requirements, or may be simply undetermined. Depending on the situation, it may be appropriate to perform in-depth analysis of the COTS software and work closely with the end user to determine which pre-existing COTS functionality is important and which functionality may interact with or be affected by the non-COTS aspects of the project. A significant regression testing effort may be needed (again, depending on the situation), and automated regression testing may be useful.

What's the best approach to software test estimation?

2007-07-07T08:33:00.003-07:00

There is no simple answer for this. The 'best approach' is highly dependent on the particular organization and project and the experience of the personnel involved.

For example, given two software projects of similar complexity and size, the appropriate test effort for one project might be very large if it was for life-critical medical equipment software, but might be much smaller for the other project if it was for a low-cost computer game. A test estimation approach that only considered size and complexity might be appropriate for one project but not for the other.

Following are some approaches to consider.

Implicit Risk Context Approach:
A typical approach to test estimation is for a project manager or QA manager to implicitly use risk context, in combination with past personal experiences in the organization, to choose a level of resources to allocate to testing. In many organizations, the 'risk context' is assumed to be similar from one project to the next, so there is no explicit consideration of risk context. (Risk context might include factors such as the organization's typical software quality levels, the software's intended use, the experience level of developers and testers, etc.) This is essentially an intuitive guess based on experience.

Metrics-Based Approach:
A useful approach is to track past experience of an organization's various projects and the associated test effort that worked well for projects. Once there is a set of data covering characteristics for a reasonable number of projects, then this 'past experience' information can be used for future test project planning. (Determining and collecting useful project metrics over time can be an extremely difficult task.) For each particular new project, the 'expected' required test time can be adjusted based on whatever metrics or other information is available, such as function point count, number of external system interfaces, unit testing done by developers, risk levels of the project, etc. In the end, this is essentially 'judgement based on documented experience', and is not easy to do successfully.

Test Work Breakdown Approach:
Another common approach is to decompose the expected testing tasks into a collection of small tasks for which estimates can, at least in theory, be made with reasonable accuracy. This of course assumes that an accurate and predictable breakdown of testing tasks and their estimated effort is feasible. In many large projects, this is not the case. For example, if a large number of bugs are being found in a project, this will add to the time required for testing, retesting, bug analysis and reporting. It will also add to the time required for development, and if development schedules and efforts do not go as planned, this will further impact testing.

Iterative Approach:
In this approach for large test efforts, an initial rough testing estimate is made. Once testing begins, a more refined estimate is made after a small percentage (eg, 1%) of the first estimate's work is done. At this point testers have obtained additional test project knowledge and a better understanding of issues, general software quality, and risk. Test plans and schedules can be refactored if necessary and a new estimate provided. Then a yet-more-refined estimate is made after a somewhat larger percentage (eg, 2%) of the new work estimate is done. Repeat the cycle as necessary/appropriate.

Percentage-of-Development Approach:
Some organizations utilize a quick estimation method for testing based on the estimated programming effort. For example, if a project is estimated to require 1000 hours of programming effort, and the organization normally finds that a 40% ratio for testing is appropriate, then an estimate of 400 hours for testing would be used. This approach may or may not be useful depending on the project-to-project variations in risk, personnel, types of applications, levels of complexity, etc.

Successful test estimation is a challenge for most organizations, since few can accurately estimate software project development efforts, much less the testing effort of a project. It is also difficult to attempt testing estimates without first having detailed information about a project, including detailed requirements, the organization's experience with similar projects in the past, and an understanding of what should be included in a 'testing' estimation for a project (functional testing? unit testing? reviews? inspections? load testing? security testing?)

With agile software development approaches, test effort estimations may be unnecessary if pure test-driven development is utilized. However, it is not uncommon to have a mix of some automated positive-type unit tests, along with some type of separate manual or automated functional testing. In general, agile-based projects by their nature will not be heavily dependent on large testing efforts, since they emphasize the construction of releasable software in very short iteration cycles. These smaller test effort estimates may not be as difficult and the impact of inaccurate estimates will be less severe.

For an interesting view of the problem of test estimation, see the comments on Martin Fowler's web site indicating that, for many large systems, "testing and debugging is impossible to schedule".

2007-07-07T08:33:00.001-07:00

How can it be determined if a test environment is appropriate?
This is a difficult question in that it typically involves tradeoffs between 'better' test environments and cost. The ultimate situation would be a collection of test environments that mimic exactly all possible hardware, software, network, data, and usage characteristics of the expected live environments in which the software will be used. For many software applications, this would involve a nearly infinite number of variations, and would clearly be impossible. And for new software applications, it may also be impossible to predict all the variations in environments in which the application will run. For very large, complex systems, duplication of a 'live' type of environment may be prohibitively expensive.

In reality judgements must be made as to which characteristics of a software application environment are important, and test environments can be selected on that basis after taking into account time, budget, and logistical constraints. Such judgements are preferably made by those who have the most appropriate technical knowledge and experience, along with an understanding of risks and constraints.

For smaller or low risk projects, an informal approach is common, but for larger or higher risk projects (in terms of money, property, or lives) a more formalized process involving multiple personnel and significant effort and expense may be appropriate.

In some situations it may be possible to mitigate the need for maintenance of large numbers of varied test environments. One approach might be to coordinate internal testing with beta testing efforts. Another possible mitigation approach is to provide built-in automated tests that run automatically upon installation of the application by end-users. These tests might then automatically report back information, via the internet, about the application environment and problems encountered.

Will automated testing tools make testing easier?

2007-07-07T08:32:00.003-07:00

Possibly. For small projects, the time needed to learn and implement them may not be worth it unless personnel are already familiar with the tools. For larger projects, or on-going long-term projects they can be valuable.
A common type of automated tool is the 'record/playback' type. For example, a tester could click through all combinations of menu choices, dialog box choices, buttons, etc. in an application GUI and have them 'recorded' and the results logged by a tool. The 'recording' is typically in the form of text based on a scripting language that is interpretable by the testing tool. Often the recorded script is manually modified and enhanced. If new buttons are added, or some underlying code in the application is changed, etc. the application might then be retested by just 'playing back' the 'recorded' actions, and comparing the logging results to check effects of the changes. The problem with such tools is that if there are continual changes to the system being tested, the 'recordings' may have to be changed so much that it becomes very time-consuming to continuously update the scripts. Additionally, interpretation and analysis of results (screens, data, logs, etc.) can be a difficult task. Note that there are record/playback tools for text-based interfaces also, and for all types of platforms.
Another common type of approach for automation of functional testing is 'data-driven' or 'keyword-driven' automated testing, in which the test drivers are separated from the data and/or actions utilized in testing (an 'action' would be something like 'enter a value in a text box'). Test drivers can be in the form of automated test tools or custom-written testing software. The data and actions can be more easily maintained - such as via a spreadsheet - since they are separate from the test drivers. The test drivers 'read' the data/action information to perform specified tests. This approach can enable more efficient control, development, documentation, and maintenance of automated tests/test cases.

Other automated tools can include:

     code analyzers - monitor code complexity, adherence to
                    standards, etc.

   coverage analyzers - these tools check which parts of the
                    code have been exercised by a test, and may
                    be oriented to code statement coverage,
                    condition coverage, path coverage, etc.

   memory analyzers - such as bounds-checkers and leak detectors.

   load/performance test tools - for testing client/server
                    and web applications under various load
                    levels.

   web test tools - to check that links are valid, HTML code
                    usage is correct, client-side and
                    server-side programs work, a web site's
                    interactions are secure.
                                     
   other tools - for test case management, documentation
                    management, bug reporting, and configuration
                    management, file and database comparisons, screen
                    captures, security testing, macro recorders, etc.

Test automation is, of course, possible without COTS tools. Many successful automation efforts utilize custom automation software that is targeted for specific projects, specific software applications, or a specific organization's software development environment. In test-driven agile software development environments, automated tests are often built into the software during (or preceding) coding of the application.

2007-07-07T08:32:00.001-07:00

What if an organization is growing so fast that fixed QA processes are impossible?
This is a common problem in the software industry, especially in new technology areas. There is generally no easy solution in this situation. One approach is:

Hire good people
Management should 'ruthlessly prioritize' quality issues and maintain focus on the customer
Everyone in the organization should be clear on what 'quality' means to the customer

Depending on the growth rate, it is possible that incremental self-managed team approaches may be applicable, such as 'Kaizen' methods of continuous process improvement, or the Deming-Shewhart Plan-Do-Check-Act cycle, and others.

2007-07-07T08:31:00.001-07:00

How can Software QA processes be implemented without reducing productivity?
By implementing QA processes slowly over time, using consensus to reach agreement on processes, focusing on processes that align tightly with organizational goals, and adjusting/experimenting/refactoring as an organization matures, productivity can be improved instead of stifled. Problem prevention will lessen the need for problem detection, panics and burn-out will decrease, and there will be improved focus and less wasted effort. At the same time, attempts should be made to keep processes simple and efficient, avoid a 'Process Police' mentality, minimize paperwork, promote computer-based processes and automated tracking and reporting, minimize time required in meetings, and promote training as part of the QA process. However, no one - especially talented technical types - likes rules or bureaucracy, and in the short run things may slow down a bit. A typical scenario would be that more days of planning, reviews, and inspections will be needed, but less time will be required for late-night bug-fixing and handling of irate customers.

Other possibilities include incremental self-managed team approaches such as 'Kaizen' methods of continuous process improvement, the Deming-Shewhart Plan-Do-Check-Act cycle, and others.

Does every software project need testers?

2007-07-07T08:30:00.001-07:00

While all projects will benefit from testing, some projects may not require independent test staff to succeed.

Which projects may not need independent test staff? The answer depends on the size and context of the project, the risks, the development methodology, the skill and experience of the developers, and other factors. For instance, if the project is a short-term, small, low risk project, with highly experienced programmers utilizing thorough unit testing or test-first development, then test engineers may not be required for the project to succeed.

In some cases an IT organization may be too small or new to have a testing staff even if the situation calls for it. In these circumstances it may be appropriate to instead use contractors or outsourcing, or adjust the project management and development approach (by switching to more senior developers and agile test-first development, for example). Inexperienced managers sometimes gamble on the success of a project by skipping thorough testing or having programmers do post-development functional testing of their own work, a decidedly high risk gamble.

For non-trivial-size projects or projects with non-trivial risks, a testing staff is usually necessary. As in any business, the use of personnel with specialized skills enhances an organization's ability to be successful in large, complex, or difficult tasks. It allows for both a) deeper and stronger skills and b) the contribution of differing perspectives. For example, programmers typically have the perspective of 'what are the technical issues in making this functionality work?'. A test engineer typically has the perspective of 'what might go wrong with this functionality, and how can we ensure it meets expectations?'. Technical people who can be highly effective in approaching tasks from both of those perspectives are rare, which is why, sooner or later, organizations bring in test specialists.

2007-07-07T08:29:00.000-07:00

What are some recent major computer system failures caused by software bugs?

A September 2006 news report indicated problems with software utilized in a state government's primary election, resulting in periodic unexpected rebooting of voter checkin machines, which were separate from the electronic voting machines, and resulted in confusion and delays at voting sites. The problem was reportedly due to insufficient testing.
In August of 2006 a U.S. government student loan service erroneously made public the personal data of as many as 21,000 borrowers on it's web site, due to a software error. The bug was fixed and the government department subsequently offered to arrange for free credit monitoring services for those affected.
A software error reportedly resulted in overbilling of up to several thousand dollars to each of 11,000 customers of a major telecommunications company in June of 2006. It was reported that the software bug was fixed within days, but that correcting the billing errors would take much longer.
News reports in May of 2006 described a multi-million dollar lawsuit settlement paid by a healthcare software vendor to one of its customers. It was reported that the customer claimed there were problems with the software they had contracted for, including poor integration of software modules, and problems that resulted in missing or incorrect data used by medical personnel.
In early 2006 problems in a government's financial monitoring software resulted in incorrect election candidate financial reports being made available to the public. The government's election finance reporting web site had to be shut down until the software was repaired.
Trading on a major Asian stock exchange was brought to a halt in November of 2005, reportedly due to an error in a system software upgrade. The problem was rectified and trading resumed later the same day.
A May 2005 newspaper article reported that a major hybrid car manufacturer had to install a software fix on 20,000 vehicles due to problems with invalid engine warning lights and occasional stalling. In the article, an automotive software specialist indicated that the automobile industry spends $2 billion to $3 billion per year fixing software problems.
Media reports in January of 2005 detailed severe problems with a $170 million high-profile U.S. government IT systems project. Software testing was one of the five major problem areas according to a report of the commission reviewing the project. In March of 2005 it was decided to scrap the entire project.
In July 2004 newspapers reported that a new government welfare management system in Canada costing several hundred million dollars was unable to handle a simple benefits rate increase after being put into live operation. Reportedly the original contract allowed for only 6 weeks of acceptance testing and the system was never tested for its ability to handle a rate increase.
Millions of bank accounts were impacted by errors due to installation of inadequately tested software code in the transaction processing system of a major North American bank, according to mid-2004 news reports. Articles about the incident stated that it took two weeks to fix all the resulting errors, that additional problems resulted when the incident drew a large number of e-mail phishing attacks against the bank's customers, and that the total cost of the incident could exceed $100 million.
A bug in site management software utilized by companies with a significant percentage of worldwide web traffic was reported in May of 2004. The bug resulted in performance problems for many of the sites simultaneously and required disabling of the software until the bug was fixed.
According to news reports in April of 2004, a software bug was determined to be a major contributor to the 2003 Northeast blackout, the worst power system failure in North American history. The failure involved loss of electrical power to 50 million customers, forced shutdown of 100 power plants, and economic losses estimated at $6 billion. The bug was reportedly in one utility company's vendor-supplied power monitoring and management system, which was unable to correctly handle and report on an unusual confluence of initially localized events. The error was found and corrected after examining millions of lines of code.
In early 2004, news reports revealed the intentional use of a software bug as a counter-espionage tool. According to the report, in the early 1980's one nation surreptitiously allowed a hostile nation's espionage service to steal a version of sophisticated industrial software that had intentionally-added flaws. This eventually resulted in major industrial disruption in the country that used the stolen flawed software.
A major U.S. retailer was reportedly hit with a large government fine in October of 2003 due to web site errors that enabled customers to view one anothers' online orders.
News stories in the fall of 2003 stated that a manufacturing company recalled all their transportation products in order to fix a software problem causing instability in certain circumstances. The company found and reported the bug itself and initiated the recall procedure in which a software upgrade fixed the problems.
In August of 2003 a U.S. court ruled that a lawsuit against a large online brokerage company could proceed; the lawsuit reportedly involved claims that the company was not fixing system problems that sometimes resulted in failed stock trades, based on the experiences of 4 plaintiffs during an 8-month period. A previous lower court's ruling that "...six miscues out of more than 400 trades does not indicate negligence." was invalidated.
In April of 2003 it was announced that a large student loan company in the U.S. made a software error in calculating the monthly payments on 800,000 loans. Although borrowers were to be notified of an increase in their required payments, the company will still reportedly lose $8 million in interest. The error was uncovered when borrowers began reporting inconsistencies in their bills.
News reports in February of 2003 revealed that the U.S. Treasury Department mailed 50,000 Social Security checks without any beneficiary names. A spokesperson indicated that the missing names were due to an error in a software change. Replacement checks were subsequently mailed out with the problem corrected, and recipients were then able to cash their Social Security checks.
In March of 2002 it was reported that software bugs in Britain's national tax system resulted in more than 100,000 erroneous tax overcharges. The problem was partly attributed to the difficulty of testing the integration of multiple systems.
A newspaper columnist reported in July 2001 that a serious flaw was found in off-the-shelf software that had long been used in systems for tracking certain U.S. nuclear materials. The same software had been recently donated to another country to be used in tracking their own nuclear materials, and it was not until scientists in that country discovered the problem, and shared the information, that U.S. officials became aware of the problems.
According to newspaper stories in mid-2001, a major systems development contractor was fired and sued over problems with a large retirement plan management system. According to the reports, the client claimed that system deliveries were late, the software had excessive defects, and it caused other systems to crash.
In January of 2001 newspapers reported that a major European railroad was hit by the aftereffects of the Y2K bug. The company found that many of their newer trains would not run due to their inability to recognize the date '31/12/2000'; the trains were started by altering the control system's date settings.
News reports in September of 2000 told of a software vendor settling a lawsuit with a large mortgage lender; the vendor had reportedly delivered an online mortgage processing system that did not meet specifications, was delivered late, and didn't work.
In early 2000, major problems were reported with a new computer system in a large suburban U.S. public school district with 100,000+ students; problems included 10,000 erroneous report cards and students left stranded by failed class registration systems; the district's CIO was fired. The school district decided to reinstate it's original 25-year old system for at least a year until the bugs were worked out of the new system by the software vendors.
A review board concluded that the NASA Mars Polar Lander failed in December 1999 due to software problems that caused improper functioning of retro rockets utilized by the Lander as it entered the Martian atmosphere.
In October of 1999 the $125 million NASA Mars Climate Orbiter spacecraft was believed to be lost in space due to a simple data conversion error. It was determined that spacecraft software used certain data in English units that should have been in metric units. Among other tasks, the orbiter was to serve as a communications relay for the Mars Polar Lander mission, which failed for unknown reasons in December 1999. Several investigating panels were convened to determine the process failures that allowed the error to go undetected.
Bugs in software supporting a large commercial high-speed data network affected 70,000 business customers over a period of 8 days in August of 1999. Among those affected was the electronic trading system of the largest U.S. futures exchange, which was shut down for most of a week as a result of the outages.
In April of 1999 a software bug caused the failure of a $1.2 billion U.S. military satellite launch, the costliest unmanned accident in the history of Cape Canaveral launches. The failure was the latest in a string of launch failures, triggering a complete military and industry review of U.S. space launch programs, including software integration and testing processes. Congressional oversight hearings were requested.
A small town in Illinois in the U.S. received an unusually large monthly electric bill of $7 million in March of 1999. This was about 700 times larger than its normal bill. It turned out to be due to bugs in new software that had been purchased by the local power company to deal with Y2K software issues.
In early 1999 a major computer game company recalled all copies of a popular new product due to software problems. The company made a public apology for releasing a product before it was ready.
The computer system of a major online U.S. stock trading service failed during trading hours several times over a period of days in February of 1999 according to nationwide news reports. The problem was reportedly due to bugs in a software upgrade intended to speed online trade confirmations.
In April of 1998 a major U.S. data communications network failed for 24 hours, crippling a large part of some U.S. credit card transaction authorization systems as well as other large U.S. bank, retail, and government data systems. The cause was eventually traced to a software bug.
January 1998 news reports told of software problems at a major U.S. telecommunications company that resulted in no charges for long distance calls for a month for 400,000 customers. The problem went undetected until customers called up with questions about their bills.
In November of 1997 the stock of a major health industry company dropped 60% due to reports of failures in computer billing systems, problems with a large database conversion, and inadequate software testing. It was reported that more than $100,000,000 in receivables had to be written off and that multi-million dollar fines were levied on the company by government agencies.
A retail store chain filed suit in August of 1997 against a transaction processing system vendor (not a credit card company) due to the software's inability to handle credit cards with year 2000 expiration dates.
In August of 1997 one of the leading consumer credit reporting companies reportedly shut down their new public web site after less than two days of operation due to software problems. The new site allowed web site visitors instant access, for a small fee, to their personal credit reports. However, a number of initial users ended up viewing each others' reports instead of their own, resulting in irate customers and nationwide publicity. The problem was attributed to "...unexpectedly high demand from consumers and faulty software that routed the files to the wrong computers."
In November of 1996, newspapers reported that software bugs caused the 411 telephone information system of one of the U.S. RBOC's to fail for most of a day. Most of the 2000 operators had to search through phone books instead of using their 13,000,000-listing database. The bugs were introduced by new software modifications and the problem software had been installed on both the production and backup systems. A spokesman for the software vendor reportedly stated that 'It had nothing to do with the integrity of the software. It was human error.'
On June 4 1996 the first flight of the European Space Agency's new Ariane 5 rocket failed shortly after launching, resulting in an estimated uninsured loss of a half billion dollars. It was reportedly due to the lack of exception handling of a floating-point error in a conversion from a 64-bit integer to a 16-bit signed integer.
Software bugs caused the bank accounts of 823 customers of a major U.S. bank to be credited with $924,844,208.32 each in May of 1996, according to newspaper reports. The American Bankers Association claimed it was the largest such error in banking history. A bank spokesman said the programming errors were corrected and all funds were recovered.
Software bugs in a Soviet early-warning monitoring system nearly brought on nuclear war in 1983, according to news reports in early 1999. The software was supposed to filter out false missile detections caused by Soviet satellites picking up sunlight reflections off cloud-tops, but failed to do so. Disaster was averted when a Soviet commander, based on what he said was a '...funny feeling in my gut', decided the apparent missile attack was a false alarm. The filtering software code was rewritten.

What is 'Software Testing'?

2007-07-07T08:28:00.001-07:00

Testing involves operation of a system or application under controlled conditions and evaluating the results (eg, 'if the user is in interface A of the application while using hardware B, and does C, then D should happen'). The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn't or things don't happen when they should. It is oriented to 'detection'.

Organizations vary considerably in how they assign responsibility for QA and testing. Sometimes they're the combined responsibility of one group or individual. Also common are project teams that include a mix of testers and developers who work closely together, with overall QA processes monitored by project managers. It will depend on what best fits an organization's size and business structure.

What is 'Software Quality Assurance'?

2007-07-07T08:27:00.001-07:00

Software QA involves the entire software development PROCESS - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented to 'prevention'.

2007-07-07T07:52:00.003-07:00

What is the purpose of the testing?

Software testing is the process used to help identify the Correctness, Completeness, Security and Quality of the developed Computer Software.

Software Testing is the process of executing a program or system with the intent of finding errors.

2007-07-07T07:52:00.001-07:00

What is quality assurance?

2007-07-07T07:50:00.003-07:00

What is the difference between QA and testing?

Testing involves operation of a system or application under controlled conditions and evaluating the results. It is oriented to 'detection'.

2007-07-07T07:50:00.001-07:00

Describe the difference between validation and verification

Verification is done by frequent evaluation and meetings to appraise the documents, policy, code, requirements, and specifications. This is done with the checklists, walkthroughs, and inspection meetings.

Validation is done during actual testing and it takes place after all the verifications are being done.

2007-07-07T07:49:00.003-07:00

Describe the Software Development Life Cycle

It includes aspects such as initial concept, requirements analysis, functional design, internal design, documentation planning, test planning, coding, document preparation, integration, testing, maintenance, updates, retesting, phase-out, and other aspects.

2007-07-07T07:49:00.001-07:00

How do you decide when you have 'tested enough’?

Common factors in deciding when to stop are:

Deadlines (release deadlines, testing deadlines, etc.)
Test cases completed with certain percentage passed
Test budget depleted
Coverage of code/functionality/requirements reaches a specified point
Bug rate falls below a certain level
Beta or alpha testing period ends

2007-07-07T07:48:00.005-07:00

What software testing types can be considered?

Black box testing – This type of testing doesn’t require any knowledge of the internal design or coding. These Tests are based on the requirements and functionality.

White box testing – This kind of testing is based on the knowledge of internal logic of a particular application code. The Testing is done based on the coverage of code statements, paths, conditions.

Unit testing – the 'micro' scale of testing; this is mostly used to test the particular functions or code modules. This is typically done by the programmer and not by testers; it requires detailed knowledge of the internal program design and code. It cannot be done easily unless the application has a well-designed architecture with tight code; this type may require developing test driver modules or test harnesses.

Sanity testing or Smoke testing – This type of testing is done initially to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing the systems in every 5 minutes or corrupting databases, the software may not be in a 'sound’ condition to proceed for further testing in its current state.

Functional testing – This a commonly used black-box testing geared to check the functional requirements of an application; this type of testing should be done by testers.

Integration testing – This testing is combining the ‘parts’ of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to the client/server and distributed systems.

Incremental Integration testing – This is continuous testing of an application when a new functionality is added the existing ones; it checks the application functionality by verifying whether it works separately before all parts of the program are completed, in this type it will be checked whether to introduce test drivers or not; this is done by programmers or by testers.

Regression testing – This is testing the whole application again after the fixes or the modifications are done on the software. This is mostly done at the end of the Software development life cycle. Mostly Automated testing tools are used for these type of testing.

System testing – This is a type of black-box type testing that is based on overall requirements specifications; covers all combined parts of a system.

End-to-end testing – This is similar to system testing; this involves testing of a complete application environment such as interacting with a database, using network communications, or interacting with other hardware, applications and so on.

UAT ( User Acceptance Testing ) – This type of testing comes on the final stage and mostly done on the specifications of the end-user or client.

Usability testing – This testing is done to check the 'user-friendliness' of the application. This depends on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers.

Compatibility testing – Testing how well the software performs in a particular hardware, software, operating system, network etc.

Comparison testing – This is nothing comparing the software strengths and weakness with another competing product.

Mutation testing – This is another method for determining if a set of test data or test cases is useful, by purposely introducing various code changes or bugs and retesting with the original test data or cases to determine whether the 'bugs' are detected.

2007-07-07T07:48:00.003-07:00

Why does software have bugs?

Miscommunication or no communication – about the details of what an application should or shouldn't do
Programming errors – in some cases the programmers can make mistakes.
Changing requirements – there are chances of the end-user not understanding the effects of changes, or may understand and request them anyway to redesign, rescheduling of engineers, effects of other projects, work already completed may have to be redone or thrown out.
Time force - preparation of software projects is difficult at best, often requiring a lot of guesswork. When deadlines are given and the crisis comes, mistakes will be made.

2007-07-07T07:48:00.001-07:00

What are the common problems in the software development process?

Inadequate requirements from the Client - if the requirements given by the client is not clear, unfinished and not testable, then problems may come.
Unrealistic schedules – Sometimes too much of work is being given to the developer and ask him to complete in a Short duration, then the problems are unavoidable.
Insufficient testing – The problems can arise when the developed software is not tested properly.
Given another work under the existing process – request from the higher management to work on another project or task will bring some problems when the project is being tested as a team.
Miscommunication – in some cases, the developer was not informed about the Clients requirement and expectations, so there can be deviations.

2007-07-07T07:47:00.006-07:00

What are the basic solutions for the software development problems?

Basic requirements - clear, detailed, complete, achievable, testable requirements has to be developed. Use some prototypes to help pin down requirements. In nimble environments, continuous and close coordination with customers/end-users is needed.
Schedules should be realistic - enough time to plan, design, test, bug fix, re-test, change, and document in the given schedule.
Adequate testing – testing should be started early, it should be re-tested after the bug fixed or changed, enough time should be spend for testing and bug-fixing.
Proper study on initial requirements – be ready to look after more changes after the development has begun and be ready to explain the changes done to others. Work closely with the customers and end-users to manage expectations. This avoids excessive changes in the later stages.
Communication – conduct frequent inspections and walkthroughs in appropriate time period; ensure that the information and the documentation is available on up-to-date if possible electronic. More emphasize on promoting teamwork and cooperation inside the team; use prototypes and proper communication with the end-users to clarify their doubts and expectations.

What is UAT testing? When it is to be done?

2007-07-07T07:47:00.005-07:00

UAT testing - UAT stands for 'User acceptance Testing. This testing is carried out with the user perspective and it is usually done before the release.

2007-07-07T07:47:00.003-07:00

What is Re- test ? What is Regression Testing ?

Re- test - Retesting means we testing only the certain part of an application again and not considering how it will effect in the other part or in the whole application.

Regression Testing - Testing the application after a change in a module or part of the application for testing that is the code change will affect rest of the application.

2007-07-07T07:47:00.001-07:00

What is the difference between structural and functional testing?

Structural testing is a "white box" testing and it is based on the algorithm or code.

Functional testing is a "black box" (behavioral) testing where the tester verifies the functional specification.