 |
Australian Digital Theses Program
|
| Thesis Details |
 Title | A Framework for Statistical Testing of
Software Components |
|
Author | Shukla, Rakeshkumar Yogendrakumar |
|
Institution | University of Queensland |
|
Date | 2007 |
|
Abstract | The term “statistical testing” is used when statistics is applied to software testing.
Statistics provides sampling theory, a sound theoretical foundation for statistical testing.
However, two difficult and important issues must be addressed in statistical testing.
The first issue is determining an accurate operational profile. Statistical testing of a
software component from a user’s point of view depends largely on the manner in which
the component is used. Characterisation of the population of expected use is referred to as
an operational profile. An operational profile is a set of input events and their associated
probabilities of occurrence expected in actual operation. The test cases that are executed
during a statistical test are a sample from the operational profile. As such, accurate
operational profiles are a critical part of statistical testing.
Determining an operational profile for the generation of test cases that are representative
of actual usage is still a difficult issue despite the considerable research in this area in
recent years. Developing an accurate operational profile for software is difficult in
general and it is very difficult for many software components because it requires
anticipating the future use of the component. The usage behaviour is typically modelled
by either Markov models or finite state machines. Previous work has focused on
exploring the occurrence of operations and has ignored parameters for operations. This
thesis presents a systematic method for deriving operational profiles for software
components that uses probabilistic statecharts to model operational profiles and that
addresses the issue of parameters for operations.
The second issue is development of a test oracle for output evaluation. Test output
evaluation is also a difficult and important problem for statistical testing. Most of the
potential benefits of testing will be lost if the success or failure of test cases is not
assessed. An expected result is needed for each test case to check the test output. The
mechanism used for test output evaluation is called a test oracle. A test oracle is an
essential part of statistical testing, because a large number of test cases is required to
represent the operational usage and the behaviour must be checked for every test case. Test result evaluation using a test oracle is widely acknowledged in the software testing
literature as a critical aspect of the testing process. Several methods for developing test
oracles, such as those using specifications and documentation, have been reported.
Unfortunately development and maintenance of such resources may require considerable
effort. A limitation in using such a resource is that the test oracle is only as good as the
resource from which it was derived. To address these issues, this thesis presents a
technique to develop a test oracle that uses the component to check its own behaviour for
test output evaluation.
While a number of proposals for statistical testing of software have been made, the
execution of test cases and test output evaluation are often not addressed in these
proposals. This thesis presents a conceptual framework for statistical testing of software
components that supports both test case execution and output evaluation.
The framework proposed in this thesis is supported by a prototype tool for test case
generation, test case execution and output evaluation. The tool supports a wide range of
operational profile approaches for test case generation and a variety of test oracles for
output evaluation. This thesis demonstrates the practical viability, flexibility and
scalability of the framework and tool support by applying them on several case studies,
including a third-party component. Experience with the framework indicates that it can
be used successfully for small to medium-sized components, including third-party
components such as COTS components.
The overall effectiveness of the statistical testing process depends on both the accuracy of
the operational profile and the effectiveness of the test oracle to detect and report the
errors encountered during testing. This thesis presents an empirical evaluation of
different types of operational profiles to determine how accurately the test cases
generated from the operational profiles represent the actual usage of the component and
different types of test oracles to determine their fault-detection ability. |
| Thesis UQ only
|
01front_shukla.pdf 202.2 Kb
02content_shukla.pdf 2487.7 Kb
|
|
