Download PUB517 Usability problem reports

43
on identifying problems and consistency across analysts is not very likely
to be improved by using specific analysis methods;
(2) re-analysis of the same data is very likely to lead to a slightly higher
thoroughness in the second analysis even if the second analysis is done
one or two months later (in this case in the range of 1-4%);
(3) within-analyst consistency does seem to be sensitive to the used analysis
method; using the SlimDEVAN method in subsequent analyses led to
within-analyst consistencies of 63-72%, whereas within-analyst
consistencies of labs using different methods in their first and second
analyses were, on average 50%.
Implications
In usability studies that are based on comparing problems, it is important
that all data are analyzed by the same analysts. This is important because of
the inherent subjectivity in extracting usability problems from observations.
With appropriate methods that conform to the characteristics as present in
SlimDEVAN and as mentioned in section 5.1, within-analyst consistency can
be improved. Also, such methods make the findings inspectable. Inspectability
is important for laying bare subjective parts in data analyses and for making
sure that outside reviewers can falsify or confirm the findings. In comparative
usability studies inspectability is also important for a thorough understanding
of the reported problems. This is crucial for being able to decide on similarity
of problems.
REFERENCES
Barendregt, W., & Bekker, M.M. (2005). Developing a coding scheme for
detecting usability and fun problems in computer games for young
children. Proceedings of Measuring Behaviour 2005, Wageningen,
The Netherlands.
Cockton G., Lavery D. & Woolrych A. (2002). Inspection-based evaluations.
In J. Jacko & A. Sears (Eds.) The human-computer interaction
handbook: fundamentals, evolving technologies and emerging
applications, (pp. 1118 - 1138) Mahwah, NJ: Lawrence Erlbaum
Associates.
Daams, B. & Hariandja J. (2006). Cross-cultural usability, testing a food
processor in Indonesia and the Netherlands. Proceedings of IEA2006:
16th World Congress on Ergonomics IEA2006 , 1-5, Amsterdam,
The Netherlands: Elsevier.