Testing

Bread Crumbs: Home - SW_Dev_Proc - Tst
Testing along with code reviews verify the correct functioning of the components according to the design as well as valid functioning of the components according to the requirements and system model.

At this point in this software development process, code reviews should have already been performed and all defects found in those reviews should now be fixed. So, this software development process has entered the testing phase and this page will focus on the testing phase.

The three testing sub-phases are module testing, sub-system integration testing, system testing, and acceptance testing. This software development process groups unit testing with other implementation activities in the implementation phase. Unit testing is typically highly coupled to code development activities with high frequency iterations and is therefore difficult to distill from the coding activities.

The following testing topics will be covered on this page:
Testing
Testing Activity PNG Module testing is the testing of complete code objects as produced by the compiler when built from source. Sub-system integration testing focuses on testing the external APIs (Application Programming Interfaces) between sub-systems. Implementation is the creation of source code from the design artifacts. Implementation involves peer reviewing source code and addressing all review findings. Implementation also requires the use of build tools to create binary files that are executable on the target. Implementation also encompasses unit-testing the source code as it is being developed and fixing defects as they are discovered. The system testing procedure tests for errors resulting from unexpected interactions among sub-systems and system components.

This page will now focus on general testing concepts that apply to varying extents to all forms of software testing including unit testing, module testing, sub-system integration testing, system testing, and acceptance testing.

Requirements Traceability

The ultimate measure of whether or not a product passes its tests is that product's ability to satisfy the requirements specified by the customer. As part of project planning, test activities were delineated. These testing tasks, in order to be useful, must ensure that the system under development fulfills its requirements and that requires traceability to those requirements. The following table provides requirements traceability for the example microwave oven project.

Task

Description

Requirements Tested

Task 1

Unit testing – Test individual components of each module

All requirements.

Task 2

Monitor and Process Inputs Module

F1 - F6, F4.1.1 – F4.1.6, F4.2.1 – F4.2.3, F4.2.7 – F4.2.10, F4.3.1, F4.3.3, F4.4.1 – F4.4.6, F4.5.1 – F4.5.4, F4.5.8.a, and F4.5.12.

Task 3

Clock Module

F4.1.4, and F4.1.6 – F4.1.9

Task 4

Timer Module

F4.2.4 – F4.2.8

Task 5

Auto Cook Presets Module

F4.3.2, and F4.3.3.a

Task 6

Auto Defrost Module

F4.4.6

Task 7

Cooking Control Module

F4.3.3.b, and F4.5.5 – F4.5.11

Test 8

Oven Control Module

F4.5.6, F4.5.7.a, and F4.5.11.a

Task 9

Display Time Module

F4.1.4, F4.1.7, F4.1.6, F4.3.3.b, F4.5.5, F4.5.7.c, and F4.5.8.d

Task 10

Display Power Module

F4.5.1

Task 11

Beeper Module

F1, F2, F4.2.5, F4.2.9, F4.5.6, F4.5.8.b, F4.5.10.b, and F4.5.11.b

Task 12

Command Processing Sub-system

F1 – F5, F4.1.1 – F4.1.6, F4.2.1 – F4.2.10, F4.3.1, F4.3.3, F4.4.1 – F4.4.6, F4.5.1 – F4.5.4, F4.5.8.a, and F4.5.12.

Task 13

Oven Control Sub-system

F4.3.3.b, and F4.5.5 – F4.5.11

Task 14

Output Processing Sub-system

F1, F2, F4.1.4, F4.1.7, F4.2.5, F4.2.6, F4.2.9, F4.3.3.b, F4.5.1, F4.5.5, F4.5.6, F4.5.7.c, F4.5.8.b, F4.5.8.d, F4.5.10.b, and F4.5.11.b

Task 15

Microwave Oven System

All requirements tested.

Task 16

Acceptance Testing

All requirements tested.


Equivalence Partitions

One of the most basic principles of testing is that you want to maximize your testing time since time is often a very scarce resource on a software development project. One method of testing that can help the project gain the greatest yield from time spent in the testing phase is to employ equivalence partitioning. Equivalence partitioning groups test variations that exercise common code paths into classes. Once test variation classes are determined, then the tester can execute the upper boundary, mid-point, and lower boundary test variations in each class to avoid redundant testing. Redundant testing wastes the precious resource of time and must be avoided when created test plans.

So, returning back to the microwave oven example, the following figure illustrates how equivalence partitioning of test variations might be used to create a cost-effective test plan.
Equivalence Partitions Method PNG

The variation identifiers are of the form, V##<P|F>. The 'V' stands for variation. The 'P' stands for an expected pass since that variation tests a nominal path through the code. The 'F' stands for an expected fail since that variation tests an off-nominal or failure path through the code. A variation that is expected to pass should assert the expected success results for the variation to pass whereas a variation that is expected to fail should assert the expected failure / error results for the variation to pass. Testing error paths is very important in order to assure thorough path coverage.

What should be clear from the above figure is that testing more than one point between boundaries is redundant (tests the same path through code) and therefore of no value.


White Box Testing

White box testing is really only applicable to testers that have access to the source code. Depending on your organization's policies, all testers in all phases of testing have access to source code. More likely, it will be developers that will have source code access. If that is the case, then this method may be applicable only to the earliest stages of testing such as unit testing and module testing.

White box testing exploits knowledge of the implementation source code to create test plans that specifically exercise certain code paths. While unit testing, it may be important to verify the workings of a complex algorithm. Similarly, it may be necessary to ensure that a random failure such as an indeterminate state or data corruption is dealt with in as graceful fashion as possible so that damage control is ensured.

The following test case is an example of how white box testing might be performed. This test case relates to the microwave oven example.

Test

Step

Action

Expected Result

Actual Result

Pass/Fail

Problem Report Number

Test 1

Step 1

Set cooking time to 00:30 and push Start button.

Oven comes on and the displayed time counts down for 30 seconds to 00:00 at which time the oven shuts off and the beeper sounds

 

 

 

Test 2

Step 1

Set cooking time to 00:30 and push Start button.

Oven turns on and displayed time begins decrementing from 00:30.

 

 

 

 

Step 2

When time equals 00:15 push Stop/Clear button.

Oven shuts off with 00:15 displayed on LCD screen.

 

 

 

 

Step 3

Immediately push Start button.*

Oven turns back on and time counts down to 00:00 at which time the oven shuts off and the beeper sounds

 

 

 

Test 3

Step 1

Set cooking time to 00:30 and push Start button.

Oven turns on and displayed time begins decrementing from 00:30.

 

 

 

 

Step 2

When time equals 00:15 push Stop/Clear button.

Oven shuts off with 00:15 displayed on LCD screen.

 

 

 

 

Step 3

Wait approximately 20 seconds..

Oven remains off with displayed time of 00:15

 

 

 

 

Step 4

Push Start button.

Oven turns back on and time counts down to 00:00 at which time the oven shuts off and the beeper sounds

 

 

 

Test 4

Step 1

Set cooking time to 00:30 and push Start button.

Oven turns on and displayed time begins decrementing from 00:30.

 

 

 

 

Step 2

When time equals 00:15 push Stop/Clear button.

Oven shuts off with 00:15 displayed on LCD screen.

 

 

 

 

Step 3

Wait approximately 20 seconds..

Oven remains off with displayed time of 00:15

 

 

 

 

Step 4

Push Stop/Clear button a second time.

Oven remains off, displayed time changes to 00:00, and the beeper sounds

 

 

 

Test 5

Step 1

Set cooking time to 01:00 and push Start button.

Oven turns on and displayed time begins decrementing from 01:00.

 

 

 

 

Step 2

When time equals 00:20 open the microwave oven door.

Oven shuts off with 00:20 displayed on LCD screen.

 

 

 

 

Step 3

Wait for approximately 30 seconds.  Push Start button to ensure oven stays off with door opened.

Oven remains off with 00:20 showing on the LCD screen.

 

 

 

 

Step 4

Close oven door and immediately push Start button.**

Oven turns back on and time counts down to 00:00 at which time the oven shuts off and the beeper sounds

 

 

 

Test 6

Step 1

Set cooking time to 01:00 and push Start button.

Oven turns on and displayed time begins decrementing from 01:00.

 

 

 

 

Step 2

When time equals 00:15 open the oven door.

Oven shuts off with 00:15 displayed on LCD screen.

 

 

 

 

Step 3

Wait approximately 20 seconds.

Oven remains off with displayed time of 00:15

 

 

 

 

Step 4

Close the oven door and wait another 20 seconds.

Oven remains off with displayed time of 00:15

 

 

 

 

Step 5

Push the Start button.

Oven turns back on and time counts down to 00:00 at which time the oven shuts off and the beeper sounds

 

 

 

Test 7

Step 1

Set cooking time to 01:00 and push Start button.

Oven turns on and displayed time begins decrementing from 01:00.

 

 

 

 

Step 2

When time equals 00:15 open the oven door.

Oven shuts off with 00:15 displayed on LCD screen.

 

 

 

 

Step 3

Wait approximately 20 seconds.

Oven remains off with displayed time of 00:15

 

 

 

 

Step 4

Push Stop/Clear button.

Oven remains off, displayed time changes to 00:00, and the beeper sounds

 

 

 

Test 8

Step 1

Set cooking time to 01:00 and push Start button.

Oven turns on and displayed time begins decrementing from 01:00.

 

 

 

 

Step 2

When time equals 00:15 open the oven door.

Oven shuts off with 00:15 displayed on LCD screen.

 

 

 

 

Step 3

Immediately close the oven door and push the Start button.***

Oven turns back on and time counts down to 00:00 at which time the oven shuts off and the beeper sounds.

 

 

 

Test 9

Step 1

Set cooking time to 00:00 and push Start button.

Oven remains off, displayed time is 00:00, and the beeper sounds.

 

 

 

* Use in-circuit emulator to activate Start signal the frame after the door is opened to prevent the code from branching to node 5 (see flow graph for this example).

** Use in-circuit emulator to activate Start signal the frame after the door is opened to prevent the code from branching to node 9 (see flow graph for this example).

*** Use in-circuit emulator to branch directly to “T” edges out of nodes 7 and 8 as shown on the flow graph for this example.

Black Box Testing

Black box testing is the opposite of white box testing as implied by the name. Where white box testing benefits from knowledge of the source code, black box testing can benefit from lack of source code knowledge. The integrity of black box testing is diminished when the tester has insight into how the target code was implemented. The point of black box testing is to use only external interfaces to drive the test variations. The tester depends on the API documentation (or at least informal function signatures from the development group) to understand how to vary the stimuli to test that the responses are correct. The only "lever" that the tester has are the external functions and the only knowledge used is API / user guide documentation.

The following test case is an example of how black box testing might be performed. This test case relates to the microwave oven example.

Test

Action

Expected Result

Actual Result

Pass/

Fail

Problem Report Number

Test 1

Cooking Time

 

 

 

 

 

Test 1.1

Boundary Value

Set cooking time to 00:00 and push Start button.

Oven remains off, displayed time is 00:00, and the beeper sounds.

 

 

 

Test 1.2

Typical Value

Set cooking time to 00:30 and push Start button.

Oven turns on and displayed time begins decrementing from 00:30.  Time decrements once per second until displayed time is 00:00 at which time the oven shuts off and the beeper sounds.

 

 

 

Test 1.3

Boundary Value

Set cooking time to 00:59 and push Start button.

Oven turns on and displayed time begins decrementing from 00:59.  Time decrements once per second until displayed time is 00:00 at which time the oven shuts off and the beeper sounds.

 

 

 

Test 1.4

Boundary Value

Set cooking time to 01:00 and push Start button.

Oven turns on and displayed time begins decrementing from 01:00.  Time decrements once per second until displayed time is 00:00 at which time the oven shuts off and the beeper sounds.

 

 

 

Test 1.5

Typical Value

Set cooking time to 03:25 and push Start button.

Oven turns on and displayed time begins decrementing from 03:25.  Time decrements once per second until displayed time is 00:00 at which time the oven shuts off and the beeper sounds.

 

 

 

Test 1.6

Boundary Value

Set cooking time to 99:59 and push Start button.

Oven turns on and displayed time begins decrementing from 99:59.  Time decrements once per second until displayed time is 00:00 at which time the oven shuts off and the beeper sounds.

 

 

 

Test 2

Timer

 

 

 

 

 

Test 2.1

Boundary Value

Set timer to 00:00 and push Start button.

Oven remains off, Displays shows 00:00 and beeper immediately sounds.

 

 

 

Test 2.2

Typical Value

Set timer to 00:30 and push Start button.

Oven remains off and displayed time begins decrementing from 00:30.  Time decrements once per second until displayed time is 00:00 at which time the beeper sounds.

 

 

 

Test 2.3

Boundary Value

Set timer to 00:59 and push Start button.

Oven remains off and displayed time begins decrementing from 00:59.  Time decrements once per second until displayed time is 00:00 at which time the beeper sounds.

 

 

 

Test 2.4

Boundary Value

Set timer to 01:00 and push Start button.

Oven remains off and displayed time begins decrementing from 01:00.  Time decrements once per second until displayed time is 00:00 at which time the beeper sounds.

 

 

 

Test 2.5

Typical Value

Set timer to 49:50 and push Start button.

Oven remains off and displayed time begins decrementing from 49:50.  Time decrements once per second until displayed time is 00:00 at which time the beeper sounds.

 

 

 

Test 2.6

Boundary Value

Set timer to 99:59 and push Start button.

Oven remains off and displayed time begins decrementing from 99:59.  Time decrements once per second until displayed time is 00:00 at which time the beeper sounds.

 

 

 

Test 3

Clock

 

 

 

 

 

Test 3.1

Invalid Value

Set clock to 00:00 and push Start button.

Beeper sounds and display indicates that an invalid time value was entered.

 

 

 

Test 3.2

Boundary Value

Set clock to 01:00 and push Start button.

Time of day displayed begins incrementing once per second from 01:00.

 

 

 

Test 3.3

Typical Value

Set clock to 01:30 and push Start button.

Time of day displayed begins incrementing once per second from 01:30.

 

 

 

Test 3.4

Boundary Value

Set clock to 01:59 and push Start button.

Time of day displayed increments from 01:59 to 02:00 then continues incrementing once per second.

 

 

 

Test 3.5

Boundary Value

Set clock to 02:00 and push Start button.

Time of day displayed begins incrementing once per second from 02:00..

 

 

 

Test 3.6

Typical Value

Set clock to 05:30 and push Start button.

Time of day displayed begins incrementing once per second from 05:30.

 

 

 

Test 3.7

Boundary Value

Set clock to 12:59 and push Start button.

Time of day displayed increments from 12:59 to 01:00 then continues incrementing once per second

 

 

 

Test 3.8

Invalid Value

Set clock to 13:00 and push Start button.

Beeper sounds and display indicates that an invalid time value was entered.

 

 

 

Test 4

Power Level

 

 

 

 

 

Test 4.1

Invalid Value

Set Power Level to 0, set time to 00:30, and push Start button.

Beeper sounds and display indicates that an invalid power level value was entered.

 

 

 

Test 4.2

Boundary Value

Set Power Level to 1, set time to 00:30, and push Start button.

Oven turns on, time begins decrementing from 00:30, and the display shows that power level 1 has been selected.

 

 

 

Test 4.3

Typical Value

Set Power Level to 5, set time to 00:30, and push Start button.

Oven turns on, time begins decrementing from 00:30, and the display shows that power level 5 has been selected.

 

 

 

Test 4.4

Boundary Value

Set Power Level to 10, set time to 00:30, and push Start button.

Oven turns on, time begins decrementing from 00:30, and the display shows that power level 10 has been selected.

 

 

 

Test 4.5

Invalid Value

Set Power Level to 11, set time to 00:30, and push Start button.

Beeper sounds and display indicates that an invalid power level value was entered.

 

 

 


Static Analysis

Static analysis targets the source code file directly. Therefore, static analyzers do not need the target of analysis to be executed in binary form. Compilers are good static analyzers in there own right. It can be of great value to have more rigorous forms of static analysis that can identify problems that the compiler may miss. There exists several types of static analysis that can be performed as shown in the following table.

Static Analysis

Description

Control Flow

Identifies loops with multiple exit / entry points and unreachable code.

Data Use

Identifies un-initialized variables, variables that are never used, and redundant test conditions.

Interface

Identifies interface type errors, functions that are never called, and function results that are never used.

Information Flow

Identifies the relationships between input variables and output variables. Derivation of values are made explicit so that the conditions affecting a variable’s value are easier to discern and understand.

Path

Identifies a permutation of all possible paths of execution so that each predicate can be analyzed individually.


Dynamic Analysis

Dynamic analysis requires the execution of the program that is undergoing analysis. Where static analysis targets the source code of the program, dynamic analysis targets the process (executing form) of the program. One of the most useful types of dynamic analysis is memory profiling. Memory profilers can detect memory leaks, as well as inefficient memory usage patterns. Another useful form of dynamic analysis are code coverage analyzers. Code coverage analyzers can provide valuable feedback regarding the amount of code coverage that is being achieved with a given test suite.

One caveat regarding dynamic analyzers is that they can change the nature of the program under analysis. Dynamic analyzers are often quite intrusive and will sometimes affect the timing of programs enough to affect the results of their execution. This is a real problem. You do not want the analysis method to change the very behavior that is being analyzed.

No part of this work should be produced or used without the permission of the authors: Michael Turner and Dr. Sharon A White.