Download Product Software Quality

Numerous metrics
[ISO/IEC 9126-2] standard contains numerous metrics. For several sub-characteristics,
[ISO/IEC 9126-2] also proposes numerous characteristics. For example, for
Operability metrics the standard proposes eight metrics. In total, the standard contains
more than a hundred external quality metrics. Assessing a product with all the
proposed metrics can take months of effort per product. Authors of the standard were
aware of this fact; therefore, they proposed evaluation based on the business
objectives and the nature of the product [ISO/IEC 9126-1]. Our approach was similar
to their proposal, so we tried to conduct a survey with the software producers from
different domains in order to define which quality sub-characteristics are relevant for
various products. The survey did not provide required responses from the industry, so
we analyzed products from various domains and defined which characteristics are
relevant. Our idea was to evaluate only a set of sub-characteristics that are relevant for
the specific product software. With this method, we reduced the number of relevant
metrics and consequently the evaluation effort per product down to one week.
General metrics
Some of the metrics proposed in [ISO/IEC 9126-2] are too general. This is logical,
because the standard was designed and written to be applicable for any software
product in any domain. Evaluators should refine the metrics according to the product
they are evaluating. For example, [ISO/IEC 9126-2] proposes two security metrics: a
metric Access Auditability, which for different products has a different meaning
(although what the metric means is clear); and Access Controllability, which can be
represented with a restricted permissions feature when evaluating an office product
software such as MS Word 2003.
Similar general metrics are efficiency-related metrics such as throughput and
response time metrics. Throughput is defined by [ISO/IEC 9126-2] as number of tasks
completed over a time period. Evaluators should define which task is product specific
and measure how many of these tasks are completed within a time period. Example of
this can be found in the literature where UPC presents a metric message throughput
for mail servers; our opinion is that UPC only refines Throughput metric of [ISO/IEC
9126-2] for mail servers.
Inapplicable metrics
Part of proposed [ISO/IEC 9126-2] metrics cannot be applied, because of the
evaluating requirements and the application methods proposed by the standard.
Examples of these metrics are Usability related metrics, where [ISO/IEC 9126-2]
recommends conducting user tests. User test according to [ISO/IEC 9126-2] means
monitoring and interviewing sample users of the product software. The standard
recommends that for reliable results at least eight users should be tested. In absence of
users, the evaluator can take that role; however, the issue here is that the evaluator has
usually better IT skills than the typical application user. The relevance of the results is
also questionable in this case because only one user is participating. We evaluated
several understandability metrics, in a way that we were executing functions as
described in the product documentation and demonstration. Example of these metrics
is completeness of description and demonstration accessibility, where [ISO/IEC 9126-
42