How to test software requirements specification (SRS)?

Do you know “Most of the bugs in software are due to incomplete or inaccurate functional requirements?” The software code, doesn’t matter how well it’s written, can’t do anything if there are ambiguities in requirements.

It’s better to catch the requirement ambiguities and fix them in early development life cycle. Cost of fixing the bug after completion of development or product release is too high. So it’s important to have requirement analysis and catch these incorrect requirements before design specifications and project implementation phases of SDLC.

How to measure functional software requirement specification (SRS) documents?
Well, we need to define some standard tests to measure the requirements. Once each requirement is passed through these tests you can evaluate and freeze the functional requirements.

Let’s take an example. You are working on a web based application. Requirement is as follows:
“Web application should be able to serve the user queries as early as possible”

How will you freeze the requirement in this case?
What will be your requirement satisfaction criteria? To get the answer, ask this question to stakeholders: How much response time is ok for you?
If they say, we will accept the response if it’s within 2 seconds, then this is your requirement measure. Freeze this requirement and carry the same procedure for next requirement.

We just learned how to measure the requirements and freeze those in design, implementation and testing phases.

Now let’s take other example. I was working on a web based project. Client (stakeholders) specified the project requirements for initial phase of the project development. My manager circulated all the requirements in the team for review. When we started discussion on these requirements, we were just shocked! Everyone was having his or her own conception about the requirements. We found lot of ambiguities in the ‘terms’ specified in requirement documents, which later on sent to client for review/clarification.

Client used many ambiguous terms, which were having many different meanings, making it difficult to analyze the exact meaning. The next version of the requirement doc from client was clear enough to freeze for design phase.

From this example we learned “Requirements should be clear and consistent”

Next criteria for testing the requirements specification is “Discover missing requirements”

Many times project designers don’t get clear idea about specific modules and they simply assume some requirements while design phase. Any requirement should not be based on assumptions. Requirements should be complete, covering each and every aspect of the system under development.

Specifications should state both type of requirements i.e. what system should do and what should not.

Generally I use my own method to uncover the unspecified requirements. When I read the software requirements specification document (SRS), I note down my own understanding of the requirements that are specified, plus other requirements SRS document should supposed to cover. This helps me to ask the questions about unspecified requirements making it clearer.

For checking the requirements completeness, divide requirements in three sections, ‘Must implement’ requirements, requirements those are not specified but are ‘assumed’ and third type is ‘imagination’ type of requirements. Check if all type of requirements are addressed before software design phase.

Check if the requirements are related to the project goal.
Some times stakeholders have their own expertise, which they expect to come in system under development. They don’t think if that requirement is relevant to project in hand. Make sure to identify such requirements. Try to avoid the irrelevant requirements in first phase of the project development cycle. If not possible ask the questions to stakeholders: why you want to implement this specific requirement? This will describe the particular requirement in detail making it easier for designing the system considering the future scope.

But how to decide the requirements are relevant or not?Simple answer: Set the project goal and ask this question: If not implementing this requirement will cause any problem achieving our specified goal? If not, then this is irrelevant requirement. Ask the stakeholders if they really want to implement these types of requirements.

In short requirements specification (SRS) doc should address following:
Project functionality (What should be done and what should not)
Software, Hardware interfaces and user interface
System Correctness, Security and performance criteria
Implementation issues (risks) if any

Conclusion:
I have covered all aspects of requirement measurement. To be specific about requirements, I will summarize requirement testing in one sentence:
“Requirements should be clear and specific with no uncertainty, requirements should be measurable in terms of specific values, requirements should be testable having some evaluation criteria for each requirement, and requirements should be complete, without any contradictions”

Testing should start at requirement phase to avoid further requirement related bugs. Communicate more and more with your stakeholder to clarify all the requirements before starting project design and implementation.

Do you have any experience testing software requirements?

Test Case Point Analysis

1. INTRODUCTION

Effective software project estimation is one of the most challenging and important activity in software project life-cycle. On time project delivery cannot be achieved without an accurate and reliable estimate. Estimates play a vital role in all phases of the software development life-cycle.
There are many popular models for estimation in vogue today. But none of the models determine the efforts needed for separate phases of the SDLC. Organizations specializing in niche areas such as providing only testing services need an estimation model that can accurately determine the size of the Application Under Test (AUT) and in turn the efforts needed to execute it. Today’s e-business applications impose new challenges for software testing. Some of the commonly known challenges are complex application scenarios, extensive third party integration, crunched testing life cycles and increased security concerns. These factors inherently make testing of e-business application far more complex and critical than conventional software testing. In no other projects is performance testing more critical than in web-based projects. Underestimating a testing project leads to under-staffing it (resulting in staff burnout), under-scoping the
quality assurance effort (running the risk of low quality deliverables), and setting too short a schedule (resulting in loss of credibility as deadlines are missed). Traditionally, estimation of efforts for testing has been more of a ballpark percentage of the rest of the development life cycle phases. This approach to estimation is more prone to errors and carries a bigger risk of delaying the launch deadlines.
TCP analysis is an approach for doing an accurate estimation of functional testing projects. This approach emphasizes on key testing factors that determine the complexity of the entire testing
cycle and gives us a way of translating test creation efforts to test execution efforts, which is very useful for regression testing estimation. In other words Test Case Points is a way of representing the efforts involved in testing projects.

2. TCP METHODOLOGY

As stated above, TCP analysis generates test efforts for separate testing activities. This is essential because testing projects fall under four different models. Though in practice, most testing projects are a combination of the four execution models stated below.

Model I – Test Case Generation
This includes designing well-defined test cases. The deliverables here are only the test cases.

Model II – Automated Script Generation
This execution model includes only automating the test cases using an automated test tool. The
deliverables include the tool generated scripts

Model III – Manual Test Execution
This execution model only involves executing the test cases already designed and reporting the
defects.

Model IV – Automated Test Execution
This phase includes the execution of the automated scripts and reporting the defects.

TCP Analysis uses a 7-step process consisting of the following stages:

1. Identify Use Cases
2. Identify Test Cases
3. Determine TCP for Test Case Generation
4. Determine TCP for Automation
5. Determine TCP for Manual Execution
6. Determine TCP for Automated Execution
7. Determine Total TCP

Given below is an overview of different phases as applied to the four project execution models.Each phase is explained in detail in subsequent sections. Determine TCP for Test Case Generation To determine the TCP for Test Case Generation, first determine the complexity of the Test Cases. Some test cases may be more complex due to the inherent functionality being tested. The complexity will be an indicator of the number of TCPs for the test case.

Calculate test case generation complexity based on the factors given below.

Sl.No Test Case Generation Complexity Factors Weights
1 The number of steps in the case –
Assuming that test cases are atomic and that they test only one
condition, the number of steps will be an indicator of the complexity. 2
2 Interface with other Test Cases
This usually involves calling other test cases from this use case. Ex.
Address Book lookup from Compose Mail 1
3 Establishing a baseline result
e.g. Testing an EMI Calculator would involve validating the formulae
used.
This would typically be complex 3

Note: The above given weights are just sample weights and should not be mistaken for
benchmarks.
– Determine the complexity based on the following table:
Number of Steps Weight
<5 1
5-10 2
>10 3

* The standard weights range from 0 to 3.
0 —-> Not Applicable
1 —-> Simple
2 —-> Average
3 —-> Complex
Calculate the sum of the above weights and determine the complexity of each test case.
Calculate the Rough Test Case Points for test generation by using the table below.
Test Case Type Complexity Weights Test Case Points
Simple < 9 6
Average < 10-16 8
Complex >16 12

Calculate Test Case Points by using the below formulae
Rough TCP-G = ( # of Simple Test Cases X 6 ) + ( # of Average Test Cases X 8 ) + ( # of
Complex Test Cases X 12 )
The TCP estimates above might be affected by certain application level parameters. For
example, if the AUT is a vertical portal (e.g. Insurance), then creating test cases would involve an
understanding of the insurance business. This might increase the effort required. To factor the
impact of such parameters, we will increase the TCP by an Adjustment Factor. Some of the
Application level parameters that might have an influence on the TCP are listed in the table
below:
Sl. No. Factors Adjusted Weight
1 Design Complexity 0.1
2 Integration with other
devices such as WAP
enabled devices, etc. 0.1
1 Multi-lingual Support 0.05
4 Total Factor 0.25

The weights in the table are only indicative and will be subject to the application for which
estimation is being carried out.
Adjustment Factor = 1 + Total Factor = 1 + 0.25 = 1.25
Each of these factors is scored based on their influence on the system being counted. The
resulting score will increase the Unadjusted Test Case Point count. This calculation provides us
with the Test Case Point Generation count.
TCP-G = Rough TCP -Generation X Adjustment Factor
2.1 Determine TCP for Automation

From the list of Test Cases derived from Section 2.2, identify the test cases that are good
candidates for automation. Some test cases save a lot of effort if done manually and is not worth
automating. On the other hand, some test cases cannot be automated because the test tool does
not support the feature being tested.
Certain cases are straightforward and are quite easy to automate whereas certain cases involving
dynamic data are difficult to automate. Next, determine the test case automation complexity
based on the factors given below.
Sl.No Test Case Automation Complexity Factors Weights
1 Interface with other Test Cases
This usually involves calling other test cases from this use case. Ex.
Address Book lookup from Compose Mail 2
2 Interface with external application
This is interaction with an application that is outside the system boundary.
Ex. Credit Card validations through an independent gateway 0
3 External Application Validation
This involves testing of third party applications to validate the use case.
Ex. Checking Word, PDF reports generated by the system 0
4 Data Driven
This is usually helpful for testing the use case with positive and negative
data. Ex. User Registration 2
5 Links
# of links to be tested for broken/orphaned links. Typically, dynamic lists
like catalog items are rated as complex. 1
6 Numerical Computations
This involves validation of arithmetical calculations.
e.g. Testing an EMI Calculator would involve validating the formulae used.
This would typically be complex. 1
7 Check Points
This involves modifying the test scripts to check for validations.
Ex. Page titles, buttons, labels, null values, max char, numeric etc 1
8 Database Check Points
This involves cross checking the database values for integrity.
Ex. Check database after user registration for proper values. 0
9 UI ComponentsThese are usually Applets, ActiveX etc 0

Note: The above given weights are just sample weights and should not be mistaken for
benchmarks.
* The standard weights range from 0 to 3.
0 —-> Not Applicable
1 —-> Simple
2 —-> Average
3 —-> Complex
Calculate the sum of the above weights and determine the complexity of each test case.
From the complexity, calculate the Test Case Points for test automation by using the table below.
Test Case Type Complexity Weights Test Case Points
Simple < 9 6
Average < 10-16 8
Complex >16 12

TCP-A (Single Scenario) = ( # of Simple Test Cases X 6 ) + ( # of Average Test Cases X 8 ) + (
# of Complex Test Cases X 12 )
e-biz applications need to be tested on various configurations because they can be accessed
from anywhere and from an uncontrolled environment. A scenario is basically a combination of a
browser, operating system and hardware.
Sometimes, the application by its nature of usage might demand testing on different scenarios.
The TCPs identified above need to be modified to factor the impact of multiple scenarios.
The extra effort needed can be obtained by answering the following two questions:
Ø Can the scripts generated for a single scenario be run on multiple scenarios?
Ø Do the scripts generated for a single scenario need to be re-generated because the
automation tool does not support a scenario?
To arrive at the additional Test Case Points, we will identify the test cases (from section 3.2) that
need to be re-generated for every additional scenario.
The TCP-A (single scenario) will be added for every scenario for which regeneration is required.
For example, assume that the test case checks for an applet condition in Internet Explorer 5.0
and Windows NT and the TCP -A comes out to be 6. Further, the test tool does not support the
same script for Netscape Navigator 4.7 and Solaris. So the script needs to be regenerated. In
this case, the Total TCP -A comes out to be 12 for the 2 different scripts.
2.2 Determine TCP for Manual Execution

To determine the TCPs for Manual Execution, first calculate the manual test case execution complexity based on the factors given below.
Sl.No Manual Execution Complexity Factors Weights
1 Pre-conditions
This usually involves setting up the test data. It also includes
the steps needed before starting the execution. E.g. to test
whether the printing facility is working fine, the pre-conditions
would include opening the application, inserting a record into
the database, generating a report and then checking the
printing facility. 2
2 Steps in the Test Case
If the steps themselves are complex, the manual execution
effort will increase. Please refer to the legend below to
determine the complexity factor. 1
3 Verification
The verification itself might be complex. For example,
checking the actual result might itself involve many steps.
Let’s say in a test case, we need to check the server logs, it
might need opening up the log analyzer and verifying the
statistics. 2

Note: The above given weights are just sample weights and should not be mistaken for benchmarks.
Number of Steps Weight
<5 1
5-10 2
>10 3

The standard weights range from 0 to 3.
0 —-> Not Applicable
1 —-> Simple
2 —-> Average
3 —-> Complex
Calculate the sum of the above weights and determine the complexity of each test case.
Calculate the Test Case Points for manual execution by using the table below.
Test Case Type Complexity Weights Test Case Points
Simple < 9 6
Average < 10-16 8
Complex >16 12

TCP-ME = ( # of Simple Test Cases X 6 ) + ( # of Average Test Cases X 8 ) + ( # of Complex
Test Cases X 12 )
Test Cases need to be manually executed in all the scenarios. To arrive at the additional Test
Case Points, the TCP -ME (single scenario) will be added for every scenario for which manual
execution is required.
2.3 Determine TCP for Automated Execution

To determine the TCP for Automated Execution, calculate the automation test case execution complexity based on the factors given below.

Note: The above given weights are just sample weights and should not be mistaken for
benchmarks.
The standard weights range from 0 to 3.
0 —-> Not Applicable
1 —-> Simple
2 —-> Average
3 —-> Complex
Calculate the sum of the above weights and determine the complexity of each test case.
Calculate the Test Case Points for automated execution by using the table below.
Sl.No Test Case Complexity Factors Weights
1 Pre-conditions
This usually involves setting up the test data. It also includes
the steps needed before starting the execution. E.g. to test
whether the printing facility is working fine, the pre-conditions
would include opening the application, inserting a record into
the database, generating a report and then checking the
printing facility. 2
Test Case Type Complexity Weights Test Case Points
Simple < 9 6
Average < 10-16 8
Complex >16 12

TCP-AE = ( # of Simple Test Cases X 6 ) + ( # of Average Test Cases X 8 ) + ( # of Complex Test
Cases X 12 )
2.4 Calculate Total TCP

The Total TCP is computed by summing up the individual TCPs for Test Case Generation, Test Automation and Test Execution.
TCP-T = TCP-G + TCP-A + TCP-ME + TCP-AE
The total TCP is indicative of the size of the testing project.
3. EFFORT CALCULATION

To translate the Test Case Points into the total person months involved, based on your prior experience estimate the number of test case points per person month.

What can be done if requirements are changing continuously?

A common problem and a major headache.

Work with the project's stakeholders early on to understand how requirements might change so that alternate test plans and strategies can be worked out in advance, if possible.
It's helpful if the application's initial design allows for some adaptability so that later changes do not require redoing the application from scratch.
If the code is well-commented and well-documented this makes changes easier for the developers.
Use rapid prototyping whenever possible to help customers feel sure of their requirements and minimize changes.
The project's initial schedule should allow for some extra time commensurate with the possibility of changes.
Try to move new requirements to a 'Phase 2' version of an application, while using the original requirements for the 'Phase 1' version.
Negotiate to allow only easily-implemented new requirements into the project, while moving more difficult new requirements into future versions of the application.
Be sure that customers and management understand the scheduling impacts, inherent risks, and costs of significant requirements changes. Then let management or the customers (not the developers or testers) decide if the changes are warranted .
Balance the effort put into setting up automated testing with the expected effort required to re-do them to deal with changes.
Try to design some flexibility into automated test scripts.
Focus initial automated testing on application aspects that are most likely to remain unchanged.
Devote appropriate effort to risk analysis of changes to minimize regression testing needs.
Design some flexibility into test cases (this is not easily done; the best bet might be to minimize the detail in the test cases, or set up only higher-level generic-type test plans)

DMAIC

DMAIC is an acronym for this five-step improvement process: Define, Measure, Analyze, Improve, Control.

* Define: Define the problem, the process, and the project goals. In Six Sigma it is imperative that the problem is specifically defined. Alluding to slowing business is a poorly defined problem. Instead, the problem should be clearly established in quantitative terms. So a good Six Sigma problem definition would be a 35% decrease of net sales in the past two consecutive quarters.
* Measure: Measure and collect data that will determine the factors that have influence over the outcome of the process or procedure.
* Analyze: The data is analyzed using statistical tools to assess whether the problem is real (and solvable) or random, which makes it unsolvable within the Six Sigma framework.
* Improve: If the problem is real, the Six Sigma team identifies solutions to improve the process based on the data analysis.
* Control: Control planning, including data collection and control mechanisms, is required to ensure that the solutions can be sustained to deliver peak performance and early deviations from the target do not materialize into process defects.

DMAIC is the method utilized for existing business processes, while DMADV, which stands for Define, Measure, Analyze, Design, Verify, is used to create new processes or designs. Both are inspired by Dr. W. Edward Deming, who is considered the father of modern quality control. Dr. Deming’s Plan-Do-Check-Act Cycle, also known as the Deming or Shewhard cycle, laid the groundwork for DMAIC as a statistical and scientific method of business process improvement.

DMAIC uses the Six Sigma methodology to improve an existing business process and profitability by identifying defect opportunities. These are places in a process, procedure, or service where defects can occur. By identifying defects per million opportunities (DMPO), Six Sigma team members can eliminate errors and can accurately determine quality, which they then use as a parameter to determine a solution to a problem.

Defects Find Rate

Definition

The Defects Find Rate counts number of new defects found in a software over period of time (day, week, month).

Calculation
To be able to calculate that metric, it is important that in your defect tracking system you track:

* affected version, version of software in which this defect was found.
* defect creation date

The calculation process:

* for each version of the product, find all defects that affected the version. If you don't track version where the defect was found you can look for all defects for the product.
* group the defects by the creation date and depending on the period you would like to evaluate count number of defects created in each day, week or month.

Dimensions
It should be possible to monitor Escaped Defects Found in following dimensions:

* Affected version, to monitori metric value of any released version.
* Project/Product. To aggregated Escaped Defects Found over all released versions of the project or product.

Usage
Defect Find Rate gives a product owner an idea of how fast new defects are found in the product and therefore how much it can influence the development and release plans.
Presentation
Best presentation is a simple bar chart, where each bar shows number of escaped defects found per (day/week/month/quarter/year). Depending on your requirements you may want to create this chart for a whole company or each product.

Defect Removal Efficiency

DefinitionThe defect removal efficiency (DRE) gives a measure of the development team ability to remove defects prior to release. It is calculated as a ratio of defects resolved to total number of defects found. It is typically measured prior and at the moment of release.
Calculation
To be able to calculate that metric, it is important that in your defect tracking system you track:

* affected version, version of software in which this defect was found.
* release date, date when version was released

DRE = Number of defects resolved by the development team / total number of defects at the moment of measurement.
Dimensions
It should be possible to monitor Defect Detection Percentage in following dimensions:

* Affected version
* Priority

PresentationDRE is typically measured at the moment of version release, the best visualization is just to show current value of DRE as a number.
ExampleFor example, suppose that 100 defects were found during QA/testing stage and 84 defects were resolved by the development team at the moment of measurement. The DRE would be calculated as 84 divided by 100 = 84%

Soak Tests (Also Known as Endurance Testing)

Soak testing is running a system at high levels of load for prolonged periods of time. A soak test would normally execute several times more transactions in an entire day (or night) than would be expected in a busy day, to identify any performance problems that appear after a large number of transactions have been executed.



Also, it is possible that a system may ‘stop’ working after a certain number of transactions have been processed due to memory leaks or other defects. Soak tests provide an opportunity to identify such defects, whereas load tests and stress tests may not find such problems due to their relatively short duration.



The above diagram shows activity for a certain type of site. Each login results in an average session of 12 minutes duration with and average eight business transactions per session.

A soak test would run for as long as possible, given the limitations of the testing situation. For example, weekends are often an opportune time for a soak test. Soak testing for this application would be at a level of 550 logins per hour, using typical activity for each login.

The average number of logins per day in this example is 4,384 per day, but it would only take 8 hours at 550 per hour to run an entire days activity through the system.

By Starting a 60 hour soak test on Friday evening at 6 pm (to finish at 6am Monday morning), 33,000 logins would be put through the system, representing 7½ days of activity. Only with such a test, will it be possible to observe any degradation of performance under controlled conditions.

Some typical problems identified during soak tests are listed below:
bullet

Serious memory leaks that would eventually result in a memory crisis,
bullet

Failure to close connections between tiers of a multi-tiered system under some circumstances which could stall some or all modules of the system.
bullet

Failure to close database cursors under some conditions which would eventually result in the entire system stalling.
bullet

Gradual degradation of response time of some functions as internal data-structures become less efficient during a long test.

Apart from monitoring response time, it is also important to measure CPU usage and available memory. If a server process needs to be available for the application to operate, it is often worthwhile to record it's memory usage at the start and end of a soak test. It is also important to monitor internal memory usages of facilities such as Java Virtual Machines, if applicable.
Long Session Soak Testing

When an application is used for long periods of time each day, the above approach should be modified, because the soak test driver is not Logins and transactions per day, but transactions per active user for each user each day.

This type of situation occurs in internal systems, such as ERP and CRM systems, where users login and stay logged in for many hours, executing a number of business transactions during that time. A soak test for such a system should emulate multiple days of activity in a compacted time-frame rather than just pump multiple days worth of transactions through the system.

Long session soak tests should run with realistic user concurrency, but the focus should be on the number of transactions processed. VUGen scripts used in long session soak testing may need to be more sophisticated than short session scripts, as they must be capable of running a long series of business transactions over a prolonged period of time.
Test Duration

The duration of most soak tests is often determined by the available time in the test lab. There are many applications, however, that require extremely long soak tests. Any application that must run, uninterrupted for extended periods of time, may need a soak test to cover all of the activity for a period of time that is agreed to by the stakeholders, such as a month. Most systems have a regular maintenance window, and the time between such windows is usually a key driver for determining the scope of a soak test.

A classic example of a system that requires extensive soak testing is an air traffic control system. A soak test for such a system may have a multi-week or even multi-month duration.