Download PUB517 Usability problem reports

6
to (even inadvertently) introduce subjectivity in findings. Instead, assurances
of integrity of findings are rooted in the data themselves. In other words, the
starting point is that (at least some degree of) subjectivity is acknowledged in
data analysis and should be dealt with properly. In Guba and Lincoln's view
this means that both the original data and the processes used to compress these
data should be available to be inspected and confirmed by outside reviewers
of the study. For this they adopt a criterion of confirmability rather than
objectivity. Probably, the term inspectability would be more appropriate, as
the products and processes should be available for inspection (not only for
confirmation, but also for falsification).
Because in usability test data analyses based on extracting usability
problems the analyst forms part of the measurement instrument, Hartson,
Andre & Williges’ (2001) definition of reliability as "… a measure of the
consistency of usability testing results across different users of the UEMs
(evaluators)" (and analyst agreement in general) should be seen as primarily
dealing with issues of subjectivity/objectivity, rather than with reliability
purely. Findings from a number of studies investigating the so-called
evaluator effect seem to confirm this notion. The evaluator effect (e.g.,
Hertzum & Jacobsen, 2001; Vermeeren, van Kesteren & Bekker, 2003; Law
& Hvannberg, 2004) is the effect that if several evaluators analyze the same
video recorded user test sessions there is a considerable lack of overlap in
their findings. The findings from the studies suggest that no matter how
careful, structured and detailed the analysis is, if it involves qualitative
analyses there is a considerable subjective component in it (e.g., see
Vermeeren, van Kesteren & Bekker, 2003). This suggests that, also in case of
user test data analyses, assurances for objectivity cannot be rooted entirely in
the method used and its subjectivity will have to be dealt with somehow.
In the remainder of this article, issues of subjectivity and objectivity will
be dealt with by using measures of analyst agreement as well as by keeping in
mind Guba & Lincoln’s (1989) advice that it should always be possible to
trace back on what data the findings are based and how the data transformed
from primary observations into findings (i.e. findings should be inspectable).
1.3 Aim of the study
Aim of the study presented in this article is to investigate issues of
consistency and inspectability of data analyses and reports from usability tests
based on extracting usability problems. For that purpose usability tests were
conducted by labs in three different countries. They all tested the same
product, applying the same test protocol. Subsequently, each individual lab
analyzed their data and reported about it. After the labs had finished writing
their reports, they were asked to re-analyze their data a few weeks later, now