A common problem and a major headache.
Work with the project's stakeholders early on to understand how requirements might change so that alternate test plans and strategies can be worked out in advance, if possible.
It's helpful if the application's initial design allows for some adaptability so that later changes do not require redoing the application from scratch.
If the code is well-commented and well-documented this makes changes easier for the developers.
Use rapid prototyping whenever possible to help customers feel sure of their requirements and minimize changes.
The project's initial schedule should allow for some extra time commensurate with the possibility of changes.
Try to move new requirements to a 'Phase 2' version of an application, while using the original requirements for the 'Phase 1' version.
Negotiate to allow only easily-implemented new requirements into the project, while moving more difficult new requirements into future versions of the application.
Be sure that customers and management understand the scheduling impacts, inherent risks, and costs of significant requirements changes. Then let management or the customers (not the developers or testers) decide if the changes are warranted .
Balance the effort put into setting up automated testing with the expected effort required to re-do them to deal with changes.
Try to design some flexibility into automated test scripts.
Focus initial automated testing on application aspects that are most likely to remain unchanged.
Devote appropriate effort to risk analysis of changes to minimize regression testing needs.
Design some flexibility into test cases (this is not easily done; the best bet might be to minimize the detail in the test cases, or set up only higher-level generic-type test plans)
Thursday, October 28, 2010
Wednesday, October 27, 2010
DMAIC
DMAIC is an acronym for this five-step improvement process: Define, Measure, Analyze, Improve, Control.
* Define: Define the problem, the process, and the project goals. In Six Sigma it is imperative that the problem is specifically defined. Alluding to slowing business is a poorly defined problem. Instead, the problem should be clearly established in quantitative terms. So a good Six Sigma problem definition would be a 35% decrease of net sales in the past two consecutive quarters.
* Measure: Measure and collect data that will determine the factors that have influence over the outcome of the process or procedure.
* Analyze: The data is analyzed using statistical tools to assess whether the problem is real (and solvable) or random, which makes it unsolvable within the Six Sigma framework.
* Improve: If the problem is real, the Six Sigma team identifies solutions to improve the process based on the data analysis.
* Control: Control planning, including data collection and control mechanisms, is required to ensure that the solutions can be sustained to deliver peak performance and early deviations from the target do not materialize into process defects.
DMAIC is the method utilized for existing business processes, while DMADV, which stands for Define, Measure, Analyze, Design, Verify, is used to create new processes or designs. Both are inspired by Dr. W. Edward Deming, who is considered the father of modern quality control. Dr. Deming’s Plan-Do-Check-Act Cycle, also known as the Deming or Shewhard cycle, laid the groundwork for DMAIC as a statistical and scientific method of business process improvement.
DMAIC uses the Six Sigma methodology to improve an existing business process and profitability by identifying defect opportunities. These are places in a process, procedure, or service where defects can occur. By identifying defects per million opportunities (DMPO), Six Sigma team members can eliminate errors and can accurately determine quality, which they then use as a parameter to determine a solution to a problem.
* Define: Define the problem, the process, and the project goals. In Six Sigma it is imperative that the problem is specifically defined. Alluding to slowing business is a poorly defined problem. Instead, the problem should be clearly established in quantitative terms. So a good Six Sigma problem definition would be a 35% decrease of net sales in the past two consecutive quarters.
* Measure: Measure and collect data that will determine the factors that have influence over the outcome of the process or procedure.
* Analyze: The data is analyzed using statistical tools to assess whether the problem is real (and solvable) or random, which makes it unsolvable within the Six Sigma framework.
* Improve: If the problem is real, the Six Sigma team identifies solutions to improve the process based on the data analysis.
* Control: Control planning, including data collection and control mechanisms, is required to ensure that the solutions can be sustained to deliver peak performance and early deviations from the target do not materialize into process defects.
DMAIC is the method utilized for existing business processes, while DMADV, which stands for Define, Measure, Analyze, Design, Verify, is used to create new processes or designs. Both are inspired by Dr. W. Edward Deming, who is considered the father of modern quality control. Dr. Deming’s Plan-Do-Check-Act Cycle, also known as the Deming or Shewhard cycle, laid the groundwork for DMAIC as a statistical and scientific method of business process improvement.
DMAIC uses the Six Sigma methodology to improve an existing business process and profitability by identifying defect opportunities. These are places in a process, procedure, or service where defects can occur. By identifying defects per million opportunities (DMPO), Six Sigma team members can eliminate errors and can accurately determine quality, which they then use as a parameter to determine a solution to a problem.
Monday, October 25, 2010
Defects Find Rate
Definition
The Defects Find Rate counts number of new defects found in a software over period of time (day, week, month).
Calculation
To be able to calculate that metric, it is important that in your defect tracking system you track:
* affected version, version of software in which this defect was found.
* defect creation date
The calculation process:
* for each version of the product, find all defects that affected the version. If you don't track version where the defect was found you can look for all defects for the product.
* group the defects by the creation date and depending on the period you would like to evaluate count number of defects created in each day, week or month.
Dimensions
It should be possible to monitor Escaped Defects Found in following dimensions:
* Affected version, to monitori metric value of any released version.
* Project/Product. To aggregated Escaped Defects Found over all released versions of the project or product.
Usage
Defect Find Rate gives a product owner an idea of how fast new defects are found in the product and therefore how much it can influence the development and release plans.
Presentation
Best presentation is a simple bar chart, where each bar shows number of escaped defects found per (day/week/month/quarter/year). Depending on your requirements you may want to create this chart for a whole company or each product.
The Defects Find Rate counts number of new defects found in a software over period of time (day, week, month).
Calculation
To be able to calculate that metric, it is important that in your defect tracking system you track:
* affected version, version of software in which this defect was found.
* defect creation date
The calculation process:
* for each version of the product, find all defects that affected the version. If you don't track version where the defect was found you can look for all defects for the product.
* group the defects by the creation date and depending on the period you would like to evaluate count number of defects created in each day, week or month.
Dimensions
It should be possible to monitor Escaped Defects Found in following dimensions:
* Affected version, to monitori metric value of any released version.
* Project/Product. To aggregated Escaped Defects Found over all released versions of the project or product.
Usage
Defect Find Rate gives a product owner an idea of how fast new defects are found in the product and therefore how much it can influence the development and release plans.
Presentation
Best presentation is a simple bar chart, where each bar shows number of escaped defects found per (day/week/month/quarter/year). Depending on your requirements you may want to create this chart for a whole company or each product.
Friday, October 22, 2010
Defect Removal Efficiency
DefinitionThe defect removal efficiency (DRE) gives a measure of the development team ability to remove defects prior to release. It is calculated as a ratio of defects resolved to total number of defects found. It is typically measured prior and at the moment of release.
Calculation
To be able to calculate that metric, it is important that in your defect tracking system you track:
* affected version, version of software in which this defect was found.
* release date, date when version was released
DRE = Number of defects resolved by the development team / total number of defects at the moment of measurement.
Dimensions
It should be possible to monitor Defect Detection Percentage in following dimensions:
* Affected version
* Priority
PresentationDRE is typically measured at the moment of version release, the best visualization is just to show current value of DRE as a number.
ExampleFor example, suppose that 100 defects were found during QA/testing stage and 84 defects were resolved by the development team at the moment of measurement. The DRE would be calculated as 84 divided by 100 = 84%
Calculation
To be able to calculate that metric, it is important that in your defect tracking system you track:
* affected version, version of software in which this defect was found.
* release date, date when version was released
DRE = Number of defects resolved by the development team / total number of defects at the moment of measurement.
Dimensions
It should be possible to monitor Defect Detection Percentage in following dimensions:
* Affected version
* Priority
PresentationDRE is typically measured at the moment of version release, the best visualization is just to show current value of DRE as a number.
ExampleFor example, suppose that 100 defects were found during QA/testing stage and 84 defects were resolved by the development team at the moment of measurement. The DRE would be calculated as 84 divided by 100 = 84%
Tuesday, October 19, 2010
Soak Tests (Also Known as Endurance Testing)
Soak testing is running a system at high levels of load for prolonged periods of time. A soak test would normally execute several times more transactions in an entire day (or night) than would be expected in a busy day, to identify any performance problems that appear after a large number of transactions have been executed.
Also, it is possible that a system may ‘stop’ working after a certain number of transactions have been processed due to memory leaks or other defects. Soak tests provide an opportunity to identify such defects, whereas load tests and stress tests may not find such problems due to their relatively short duration.
The above diagram shows activity for a certain type of site. Each login results in an average session of 12 minutes duration with and average eight business transactions per session.
A soak test would run for as long as possible, given the limitations of the testing situation. For example, weekends are often an opportune time for a soak test. Soak testing for this application would be at a level of 550 logins per hour, using typical activity for each login.
The average number of logins per day in this example is 4,384 per day, but it would only take 8 hours at 550 per hour to run an entire days activity through the system.
By Starting a 60 hour soak test on Friday evening at 6 pm (to finish at 6am Monday morning), 33,000 logins would be put through the system, representing 7½ days of activity. Only with such a test, will it be possible to observe any degradation of performance under controlled conditions.
Some typical problems identified during soak tests are listed below:
bullet
Serious memory leaks that would eventually result in a memory crisis,
bullet
Failure to close connections between tiers of a multi-tiered system under some circumstances which could stall some or all modules of the system.
bullet
Failure to close database cursors under some conditions which would eventually result in the entire system stalling.
bullet
Gradual degradation of response time of some functions as internal data-structures become less efficient during a long test.
Apart from monitoring response time, it is also important to measure CPU usage and available memory. If a server process needs to be available for the application to operate, it is often worthwhile to record it's memory usage at the start and end of a soak test. It is also important to monitor internal memory usages of facilities such as Java Virtual Machines, if applicable.
Long Session Soak Testing
When an application is used for long periods of time each day, the above approach should be modified, because the soak test driver is not Logins and transactions per day, but transactions per active user for each user each day.
This type of situation occurs in internal systems, such as ERP and CRM systems, where users login and stay logged in for many hours, executing a number of business transactions during that time. A soak test for such a system should emulate multiple days of activity in a compacted time-frame rather than just pump multiple days worth of transactions through the system.
Long session soak tests should run with realistic user concurrency, but the focus should be on the number of transactions processed. VUGen scripts used in long session soak testing may need to be more sophisticated than short session scripts, as they must be capable of running a long series of business transactions over a prolonged period of time.
Test Duration
The duration of most soak tests is often determined by the available time in the test lab. There are many applications, however, that require extremely long soak tests. Any application that must run, uninterrupted for extended periods of time, may need a soak test to cover all of the activity for a period of time that is agreed to by the stakeholders, such as a month. Most systems have a regular maintenance window, and the time between such windows is usually a key driver for determining the scope of a soak test.
A classic example of a system that requires extensive soak testing is an air traffic control system. A soak test for such a system may have a multi-week or even multi-month duration.
Also, it is possible that a system may ‘stop’ working after a certain number of transactions have been processed due to memory leaks or other defects. Soak tests provide an opportunity to identify such defects, whereas load tests and stress tests may not find such problems due to their relatively short duration.
The above diagram shows activity for a certain type of site. Each login results in an average session of 12 minutes duration with and average eight business transactions per session.
A soak test would run for as long as possible, given the limitations of the testing situation. For example, weekends are often an opportune time for a soak test. Soak testing for this application would be at a level of 550 logins per hour, using typical activity for each login.
The average number of logins per day in this example is 4,384 per day, but it would only take 8 hours at 550 per hour to run an entire days activity through the system.
By Starting a 60 hour soak test on Friday evening at 6 pm (to finish at 6am Monday morning), 33,000 logins would be put through the system, representing 7½ days of activity. Only with such a test, will it be possible to observe any degradation of performance under controlled conditions.
Some typical problems identified during soak tests are listed below:
bullet
Serious memory leaks that would eventually result in a memory crisis,
bullet
Failure to close connections between tiers of a multi-tiered system under some circumstances which could stall some or all modules of the system.
bullet
Failure to close database cursors under some conditions which would eventually result in the entire system stalling.
bullet
Gradual degradation of response time of some functions as internal data-structures become less efficient during a long test.
Apart from monitoring response time, it is also important to measure CPU usage and available memory. If a server process needs to be available for the application to operate, it is often worthwhile to record it's memory usage at the start and end of a soak test. It is also important to monitor internal memory usages of facilities such as Java Virtual Machines, if applicable.
Long Session Soak Testing
When an application is used for long periods of time each day, the above approach should be modified, because the soak test driver is not Logins and transactions per day, but transactions per active user for each user each day.
This type of situation occurs in internal systems, such as ERP and CRM systems, where users login and stay logged in for many hours, executing a number of business transactions during that time. A soak test for such a system should emulate multiple days of activity in a compacted time-frame rather than just pump multiple days worth of transactions through the system.
Long session soak tests should run with realistic user concurrency, but the focus should be on the number of transactions processed. VUGen scripts used in long session soak testing may need to be more sophisticated than short session scripts, as they must be capable of running a long series of business transactions over a prolonged period of time.
Test Duration
The duration of most soak tests is often determined by the available time in the test lab. There are many applications, however, that require extremely long soak tests. Any application that must run, uninterrupted for extended periods of time, may need a soak test to cover all of the activity for a period of time that is agreed to by the stakeholders, such as a month. Most systems have a regular maintenance window, and the time between such windows is usually a key driver for determining the scope of a soak test.
A classic example of a system that requires extensive soak testing is an air traffic control system. A soak test for such a system may have a multi-week or even multi-month duration.
Subscribe to:
Posts (Atom)