This concludes the introduction to test planning. The next paragraphs discuss the common activities between Software Engineering and Test Engineering. The activities are; identifying the test phases, estimating the test effort; test tools; defect tracking; definitions and handling problems.

Identify the Test Phases
This is done very early in the planning phase and should take into account the test requirements in the contract and the risk areas of the test effort.

1. Consider the following example: the contract calls for formal qualification test (FQT), system level test, and acceptance test. Acceptance test will be conducted in the target environment after installing the hardware and software at the site. An additional test phase, installation test, would help to mitigate the risk associated with conducting acceptance test on a newly installed system.
2. The difference between Informal and Formal Test is introduced here and distinguished throughout the course.
   1. Informal Test refers to those phases of test conducted by Software Engineering or an integration test team; these are activities that are planned and conducted outside of the independent test team.
      Informal test:
   1. Unit test planned & conducted by Software Engineering
   2. Integration test planned & conducted by Software Engineering or an integration team
   3. Informal does not imply a lack of process
   2. Formal Test refers to those phases of test that are planned and conducted by the independent test team referred to as Test Engineering.
      Formal test:
      1. Planned & conducted by an independent Test Engineering team
      2. Fully documented per phase
         • Test plans
         • Test descriptions & procedures (test cases)
         • Test reports
   3. The phases identified below are addressed in this unit.

1. To ensures changes have not corrupted existing functions. Test Engineering want to make sure that in making software fixes, the Software Engineering have not corrupted functionality that was correct before.
2. To requires a set of baselined test cases. These are test cases with known results that have been baselined and previously executed. New test cases are not introduced during regression test. New test cases may be introduced if Test Engineering is doing ad hoc testing and uncovers a defect. But this is generally not the time for ad hoc testing.
3. To perform a regression test set (set of baseline tests) exercises major functional areas. During regression test, a subset of the full set of test cases are generally executed. This subset will contain test cases that uncovered defects and the fix is in the current build; test cases that look for side-effects of the code fixes; and possibly areas of concern, such as critical areas of the software.
4. To review or develop the project standards for guidelines on the scope of regression test. The project standard may supply guidelines on scope. For instance, DoD-MSD-498 (2167A) calls this "retest" and provides specific guidelines on the scope of regression test. Some Test Engineering teams operate under the philosophy that all test cases are re-run during regression test. This needs to be determined up front to plan adequate time to conduct regression test.

The other issue that may affect the schedule for regression test is what do you do if a defect is found? Do you (1) fix it and continue on with the test cases that have not yet been executed, or (2) fix it and re-run all test cases -- another full regression test? (3) Identify as a known defect and document it for the next build.

V Model of the Software Development Life Cycle
The V Model of the Software Development Life cycle depicted below illustrates a typical software development life cycle with test development and execution activities. The left side of the "V" identifies the specification, design, and coding activities for developing software. It also indicates when the test planning and design activities begin. For example, the acceptance, installation, system, and function (FQT) tests can be planned and prepared as soon as software requirements are baselined. The integration tests can be planned and prepared as soon as the software design structures are known. And the unit tests can be specified and designed as soon as the code units are prepared. The subject matter presented in this course is followed according to the way the phases are encountered in the V Model of the Software Development Life Cycle: unit, integration, function/FQT, system, installation, and acceptance. Also see Figure 2-11 page 51 IV&V by R. O. Lewis.

Click here to view animated version.

Testing Schedules and Estimating Staff
Detailed schedules for testing milestones and supporting activities are established for the entire project at test planning time. These schedules should be detailed enough to provide management information to measure test progress, but flexible enough to permit reaction to unanticipated problems. Test Engineering does this early in the test planning phase.

The testing schedules need to be developed using the software development schedule. This is because most of the testing activities depend upon the availability of software requirements, design, and code. The testing activities need to correspond to the availability of these development products.

The following list is some of the considerations used in estimating the level of effort necessary to plan and conduct each phase of test. The list is not an inclusive list of things that must be considered in estimating the level of effort necessary to plan and conduct each phase of test. These are some of the more common issues. The list is unique to each software project.

1. Contractual requirements
2. Number of requirements
3. Complexity of system, software, & interfaces
4. Lines of code
5. Security & safety considerations
6. Maturity of hardware & operating system
7. Function point analysis
8. Team structure

Test Engineering should use scheduling tools, industrial standards and metrics to estimate schedule and staff. The scheduling tool may be dictated by the contract or COTS. Use of these tools is very common for the testing stage. Test Engineering should be sure to include adequate time for tool preparation and training on the use of the tools. Industrial standard for testing time related to the specific software should be obtained. If Test Engineering has managed a test phase before they should have collected metrics on how long certain activities take. Test Engineering can then create relationships between source lines of code (SLOC), requirements, and test work products to predict the current efforts.

Scheduling Guidelines
The Scheduling Guidelines below show some general ideas as many of these activities are overlooked or underestimated in preparing schedules.

Test Engineering must consider other resources that they will ask management for during test planning. The test resources (excluding personnel) are the hardware suites, actual data, testing tools and materials.

Hardware suites resource planning should start very early. This is especially important for very large systems, where it may or may not be reasonable to obtain a full target suite (like a M1A2 Tank or an 767 airplane). The following types of suites need to be addressed in the planning stage for applicability. Software Engineering will need hardware to develop and debug code (software development suite). Test Engineering will need hardware to develop test work products on (test development suite) as well as execute test cases on (formal test suite). This latter suite should be absolutely independent of any other suite. It should consist, as nearly as possible, of all components of the customer’s ultimate configuration.

Actual (real) data is difficult, time consuming, or costly to obtain. It is suggested that test data be addressed as early as possible to allow adequate time to identify and then obtain the types of data that are needed. At times tools will need to be written to further perturb the data. The writing of these tools will need to be negotiated as to whether Software or Test Engineering will perform the design and development of these tools. Remember that Software Engineering will require some of this data during unit and integration test.

Test Design and Planning will have identified any special tools needed for testing. These might include special software, communications test hardware or other equipment. The key point is the tools must be identified, funded, and controlled. The manager(s) of unit and integration test must also plan out any stubs and drivers needed for these early phases of testing or devise a strategy for first testing stand-alone modules and then once these are baselined they may be used for testing other units/components (dependent on whether unit or integration test).

Materials includes any special documentation (requirements, design, user’s manuals, etc.) or consumable products (paper, tapes, disks, etc.) required during testing.

Program Level Reviews
Test activities are not heavily included in reviews early in the project life cycle. But there are often misunderstandings about the role of System Engineering, Software Engineering, and Test Engineering in program level reviews with respect to test-related activities. So these roles must be established early in the project life cycle to adequately plan support for these reviews. Two examples of program level reviews are critical design review and test readiness review. When planning for the program level reviews, identify the System, Software, and Test Engineering team participation with respect to test-related activities. For example, if the Software Development Plan is being reviewed, will Software Engineering or Test Engineering present the section on test?

The Test Readiness Review (TRR) represents the entrance into Formal Test. Software Engineering and Test Engineering are the major participants. Since TRR represents the entrance into Formal Test, this review concentrates on ensuring that Software Engineering has successfully completed their role and Test Engineering has successfully prepared for their role. The TRR requires coordination between all teams, not only in preparing for the actual review, but in planning, preparing for and conducting the work required by Software Engineering to end their phase and Test Engineering to begin their phase.

This lesson presented the Test Planning introduction and the common activities performed by Test Engineering and Software Engineering. This Unit also presented the Master Test Plan, test types, test schedule and staff activities and program level reviews.

Note: Please click Quiz on your left navigation and take Quiz 3.