Tool Mentor: Analyzing Test Execution Using Rational Test RealTime

Purpose

This tool mentor describes how to analyze test execution using Rational Test RealTime.

It relates to this Rational Unified Process information:

• Analyze Test Failure

Overview

Test RealTime's testing tools can generate two different types of test reports:

• The test verdict report provides an assessment of each test with a Passed or Failed status.
• The UML sequence diagram report provides a more detailed graphical test report.

Typically, you acknowledge the results provided by the test verdict report first, and then you analyze the root causes of test failure, if any, with the UML sequence diagram report.

The Coverage, PurifyLT, and QuantifyLT observation tools generate table results and the Trace tool provides a graphical UML sequence diagram of the application's execution sequence.

For executing test components, see the Tool Mentor titled Executing Test Components Using Rational Test RealTime.

Tool Steps

1. View the test verdicts
2. Evaluate your results
3. Evaluate the coverage of your tests
4. Submit a Rational ClearQuest defect report using Test RealTime

1.  View the test verdicts

The test verdict is a summary text report that tells you if the source code under test effectively passed or failed the tests.

To open a test verdict report:

• Right-click a successfully executed test node.
• From the context menu, select View and Report Viewer.

The report is organized into two main sections:

• Header — the report header shows the name of the report file, as well as the global Passed or Failed status. The statistics are calculated on the actual number of test elements listed in the Test Script section.
• Test Results — this section summarizes the results of the test script. Tests components are displayed hierarchically with their Failed, Passed, or Not Tested status.

The Report Viewer works in conjunction with the Report Browser pane. Click on an element in the Report Browser to go to the corresponding line.

For detailed information on using the Report Viewer, refer to the Rational Test RealTime User Guide.

2. Evaluate your results 

Object Testing and System Testing each generate a UML Sequence Diagram (SD) report, which provides you with the graphical detail of the test execution sequence, as well as the interaction between application components.

A UML Sequence Diagram report helps you understand why and when a test has failed. For system testing this also gives you a visual report of the messages passing between the system under test and the virtual testers.

To open the UML sequence diagram report:

• Right-click a successfully executed test node.
• From the context menu, select View and UML/SD Viewer.

Test components are displayed as green or red UML notes, depending on their Passed or Failed status.

The UML sequence diagram report works in conjunction with the Report Browser pane. The Report Browser offers a hierarchical view of instances and is a practical way to locate an element in the UML/SD Viewer.

For detailed information on using the UML/SD Viewer, refer to the Rational Test RealTime User Guide.

3. Evaluate the coverage of your tests

An essential parameter when analyzing the relevance of a test or a test campaign is the coverage rate of the source code under test. Test RealTime's Coverage tool can instrument the application under test to provide a Code Coverage report. This report marks each line of the source code as Covered or Not-covered by the test. Coverage types include control structures (iterations, selections), conditions (multiple, modified, forced), and functions (inputs, outputs, calls).

To obtain a Coverage Rate report on a test:

• Select a test node.
• In the Build toolbar, click Build Options.
• Select Coverage.
• Rerun the test node.
• Right-click the successfully executed test node.
• From the context menu, select View and Coverage Report.

For detailed information on using Coverage, refer to the Rational Test RealTime User Guide.

4. Submit a ClearQuest defect report using Test RealTime

When testing software, tracking defects and change requests is essential. Test RealTime offers seamless integration with Rational ClearQuest, a defect and change tracking (DCT) tool designed to operate in a client/server environment. ClearQuest helps you determine the quality of your application or component during each phase of the development cycle and helps you track the release where a feature, an enhancement, or a bug fix appears.

To submit a defect report for a failed test:

• Right-click on the test node that you want to add to the defect report.
• From the context menu, select Submit ClearQuest Defect. ClearQuest displays a submit form that already contains information about the selected test
• Enter any additional information about the defect and click OK to submit the defect.

Rational Unified Process