How do you know that the software product released into production is virtually defect-free? The one-word answer is simple, but definitely not easy: “testing”. Testing is often the single most overlooked aspect of a project. One of the most well known facts about software defects is that the longer they go undiscovered, the more expensive they are to fix when discovered. Testing allows for the identification of issues and defects before implementation.
General RuleThe cost of correcting a software defect grows exponentially for each downstream phase of the development lifecycle in which it remains undiscovered. A simple ratio to consider is 1:10:100. If a defect costs one unit (hour or dollar) to correct in the Define and Design phases, it costs approximately 10 units to fix in the Development and Testing phases, and over 100 times to fix in production, after the Deploy phase.
Increasing Cost of Defect Correction – Based on Historic DataThe preceding diagram describes in more detail the relative cost of a defect at different times in the life cycle of a software project, based on historic data. Initially, the cost of fixing a defect at the Requirements stage, when everything is on paper only is somewhat irrelevant. However, as the project moves along in its life cycle the cost of fixing a defect increases exponentially. At the initial Requirements stage, when a defect is no more than a change in thought, the relative cost is times 1. During Design phase, the relative cost is times 5 what it was when compared to the Requirements phase, and then times 10 what it was when it becomes code and on this goes until the relative cost of a bug fix is 150 times what it was originally. Measure twice and cut once.
Case Study – “United States of America”A major research project conducted by the United States Department of Commerce, National Institute of Standards and Technology showed that in a typical software development project, approximately 80% of the development phase budget is the result of correcting software defects. The NIST study estimated that software defects cost the U.S. economy $60 billion per year. How much does it cost your company?
How Does Testing Help?Implementing a testing strategy helps to drastically reduce the potential for defects to reach production. If proper testing is executed throughout the project lifecycle, the likelihood that a defect will be implemented into a production environment is virtually eliminated. By examining the accuracy of the developed code compared to the requirements and design, you ensure that only proper, virtually defect-free code reaches your production environment.
How Much Testing?As a good “rule of thumb”, plan on allocating 20 – 25% of your overall development budget for testing purposes. This proportional number allows for the testing of some levels of complexity to occur. If the project contains 100 hrs of development time, plan for 20-25 hrs. However, if the project has 1,000 hrs of development, 20-25 hrs will probably not be sufficient to test all levels of functionality and complexity.
Testing TypesEach project carries a specific testing need, and your testing strategy should be planned in the infant stages of the project lifecycle. For example, a small brand new informational website may not require load testing, whereas a large complex global E-commerce-to-ERP integration may require extensive load and regression testing. The Project Manager would typically determine the testing strategy in conjunction with the Stakeholder(s) upfront. In addition, the test plan for each strategy should be signed off by the Project Manager and the Stakeholder(s) prior to both executing the test plan and deploying any code to a production environment. This helps to ensure that everyone is on the same page.
This is the final set of tests to ensure that the solution meets the business objectives and requirements. The business customers are typically responsible for this testing, as it requires their active participation to work with the solution, as they will when it moves to production status.
Compatibility testing can be performed manually or can be driven by an automated functional or regression test suite. It ensures compatibility of an application or Web site with different browsers, OSs, and hardware platforms.
Producing tests for the behavior of an implementation to be sure it provides the portability, interoperability, and/or compatibility a standard defines. It also allows for verifying that the implementation conforms to industry standards.
Validating an application or web site conforms to its specifications and correctly performs all its required functions. This entails a series of tests which perform a feature by feature validation of behavior, using a wide range of normal and erroneous input data. This can involve testing of the product's user interface, APIs, database management, security, installation, networking, etc. Functional testing can be performed on an automated or manual basis using black box or white box methodologies.
Black Box:Testing without knowledge of the internal workings of the item being tested.
White Box Testing:Testing based on an analysis of internal workings and structure of a piece of software.
After the components have been unit tested, they are executed together. This testing verifies that the interfaces work correctly and that data is processed in its entirety as expected. Integration Testing follows unit testing and precedes system testing.
Load testing is a generic term covering Performance Testing and Stress Testing.
Performance Testing:Performance testing can be applied to understand your application or web site's scalability, or to benchmark the performance in an environment of third party products such as servers and middleware for potential purchase. This sort of testing is particularly useful to identify performance bottlenecks in high use applications. Performance testing generally involves an automated test suite as this allows easy simulation of a variety of normal, peak, and exceptional load conditions.
Stress Testing:Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements to determine the load under which it fails and how. A gradual degradation under load leading to non-catastrophic failure is the desired result. Often, Stress Testing is performed using the same process as Performance Testing but employing a very high level of simulated load.
Regression testing ensures that there is no impact to existing functionality, prior to implementing the new functionality. Commonly considered one of the most important testing types for new implementations onto existing systems.
A quick-and-dirty test which verifies that the major functions of a piece of software are working without attention to finer details.
System testing is where the most rigorous testing takes place, to ensure that the solution meets the technical and environmental needs of the customer and that the solution behaves as it should. Many individual tests can be performed under the system test umbrella. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code.
The first testing which occurs is to ensure that the component meets expectations in terms of features and functionality. In almost all cases, the person that develops the component also does unit testing. Unit testing is typically conducted in a local or development environment.Be sure to check out my next blog in the Project Leadership Series, including how to write effective and efficient test plans!