Roles and Activities >
Analyst Role Set >
Test Analyst >
Determine Test Results
Roles and Activities >
Analyst Role Set >
Test Analyst >
Determine Test Results
Activity:
| ||||||||||||||||||||||||||||||||||||||||
| Workflow Details: |
| Purpose: | To investigate each incident and obtain detailed understanding of the resulting problems. |
In this activity, the Test Logs are analyzed to determine the meaningful Test Results, regarding the differences between the expected results and the actual results of each test. Identify and analyze each incident and failure in turn. Learn as much as you can about each occurrence.
Check for duplicate incidents, common symptoms and other relationships between incidents. These conditions often provide valuable insight into the root cause of a group of the incidents.
| Purpose: | To enter change request information into a tracking tool for assessment, management, and resolution. |
Differences indicate potential defects in the Target Test Items and should be entered into a tracking system as incidents or Change Requests, with an indication of the appropriate corrective actions that could be taken.
Verify that there is accurate, supporting data available. Collate the data for attachment directly to the Change Request, or reference where the data can be obtained separately.
Whenever possible, verify that the problem is reproducible. Reproducible problems have much more likelihood of receiving developer attention and being subsequently fixed; a problem that cannot be reproduced both frustrates development staff and will waste valuable programming resources in fruitless research. We recommend that you still log these incidents, but that you consider identifying unreproducable incidents separately from the reproducible ones.
It's important for Change Requests to be understandable, especially the headline. Make sure the headline is crisp and concise, articulating clearly the specific issue. A brief headline is useful for summary reports and discussion in CAB status meetings.
It's important that the detailed description of the Change Request is unambiguous and can be easily interpreted. It's a good idea to log your Change Requests as soon as possible, but take time to go back and improve and expand on your descriptions before they are viewed by development staff.
Provide candidate solutions, as many as practical. This helps to reduce any remaining ambiguity in the description, often helping to clarify. It also ensures increases the likelihood that the solution will be close to your exceptions. Furthermore, it shows that the test team is not only prepared to find the problems, but also to help identify appropriate solutions.
Other details to include are screen image captures, Test Data files, automated Test Scripts, output from diagnostic utilities and any other information that would be useful to the developers in isolating and correcting the underlying fault.
Provide an indication to the management and development staff of the severity of the problem. In larger teams the actual resolution priority is normally left for the management team to determine, however you might allow individuals to indicate their preferred resolution priority and subsequently adjust as necessary. As a general rule, we recommend you assign Change Requests an average resolution priority by default, and raise or lower that priority on a case-by-case basis as necessary.
You may need to differentiate between the impact the Change Request will have on the production environment if it isn't addressed and the impact the Change Request will have on the test effort if it isn't addressed; It's just as important for the management team to know when a defect is impacting the testing effort as it is to be aware of severity to end users.
Sometimes it's difficult to see in advance why you need both attributes. It's possible that an incident may be really severe, such as a system crash, but the actions required to reproduce it very unlikely to occur. In this case the team may indicate it's severity as high, but indicate a very low resolution priority.
Incidents often bare out the old adage"Where there's smoke, there's fire"; as you identify and log one Change Request, you quite often identify other issues that need to be addressed. Avoid the temptation to simply add these additional findings to the existing Change Request: if the information is directly related and helps to solve the existing issue better, then that's OK. If the other issues are different, identifying them against an existing CR may result in those issues not being actioned, not getting appropriate priority in their own right, or impacting the speed at which other issues are addressed.
| Purpose: | To calculate and deliver the key measures and indicators of test. |
Analyze the incidents based on where they are distributed, such as functional area, quality risk, assigned tester and assigned developer.
Look for patterns in the distribution, such as functional areas that appear to have above average defects count. Also look for both developers and testers that may be overworked and where their quality of work is slipping
To evaluate test execution coverage, you need to review the Test Logs and determine:
The objective is to ensure that a sufficient number of the tests targeted for this Test Cycle have been executed usefully. If this is not possible, or to augment that execution data, one or more additional test coverage criteria can be identified, based upon:
See "Concepts: Key Measures of Test, Requirements-based test coverage".
Record an present the Test Results in an Test Evaluation Report for this Test Cycle.
To analyze defects, you need to review and analyze the measures chosen as part of your defect analysis strategy. The most common defect measures used include the following different measures (often displayed in the form of a graph):
Compare the measures from this Test Cycle to the running totals for the current Iteration and those from the analysis of previous iterations, to better understand the emerging trends over time.
It is recommended you present the results in diagram form with supporting findings on request.
| Purpose: | To give feedback on the current perceived or experienced quality in the software product. |
| Purpose: | To provide feedback on what remaining areas of risk provide the most potential exposure to the project. |
Identify and explain those areas that have not yet been addressed in terms of quality risks and indicate what impact and exposure this leaves the team.
Provide an indication of what priority you consider each outstanding quality risk to have, and use the priority to indicate the order in which these issues should be addressed.
| Purpose: | To make a summary assessment of the test coverage analysis. |
Based on the work in step test execution coverage, provide a brief summary statement of the status and information the data represents.
| Purpose: | To communicate the results of testing to stakeholders and make an objective assessment of quality and test status. |
This step in the activity is to generate the Test Evaluation Summary. This is accomplished by assembling the above information into a single report and distributing it to the appropriate stakeholders for review.
Record and present the Test Results in an Test Evaluation Summary for this Test Cycle.
| Purpose: | To promote and publicize the Evaluation Summary as appropriate. |
| Purpose: | To verify that the activity has been completed appropriately and that the resulting artifacts are acceptable. |
Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and that it is complete enough to be useful to those team members who will make subsequent use of it as input to their work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough".
Have the people performing the downstream activities that rely on your work as input take part in reviewing your interim work. Do this while you still have time available to take action to address their concerns. You should also evaluate your work against the key input artifacts to make sure you have represented them accurately and sufficiently. It may be useful to have the author of the input artifact review your work on this basis.
Try to remember that that RUP is an iterative process and that in many cases artifacts evolve over time. As such, it is not usually necessary—and is often counterproductive—to fully-form an artifact that will only be partially used or will not be used at all in immediately subsequent work. This is because there is a high probability that the situation surrounding the artifact will change—and the assumptions made when the artifact was created proven incorrect—before the artifact is used, resulting in wasted effort and costly rework. Also avoid the trap of spending too many cycles on presentation to the detriment of content value. In project environments where presentation has importance and economic value as a project deliverable, you might want to consider using an administrative resource to perform presentation tasks.
|
Rational Unified
Process
|
