Download ACRES 3 User Guide

Transcript
recall =
2 × precision × recall
TP
TP
F _ measure =
precision =
precision + recall
TP + FP ,
TP + FN ,
Finally the weighted average of these metrics for all classes can be calculated, taking into account
the number of occurrences of each class in the dataset.
These metrics are widely used in classification performance evaluations and corresponding tools
like the data mining tool Weka so using them allows the direct comparison with various
classification models.
Evaluation Report in ACRES
The dataset is partitioned in two sets: training
and testing set. The expert system is generated
using the training set and then it is evaluated
on the data of the testing set. The procedure is
repeated for different partitions of the dataset
(cross validation) and the average values of
the metrics are presented.
The produced expert system does not simply
classify an instance to the predicted class. It
provides an uncertainty value for each
possible class. In order to make the evaluation
of a produced expert system
easier, we consider that the
system classifies an instance
to the class for which the
uncertainty factor is the
highest.
As described above, to
evaluate an expert system for
more than two classes, we
evaluate the performance for
each class separately. For each class i, we treat the problem as binary, with the first class being i
and the second class being a class consisting of all other classes. We then form a confusion matrix
and compute the metrics for each class i.
We are mostly interested in the Sensitivity and Precision metrics. We also combine these two
metrics producing their mean proportional, SQRT(p*r) as a more general metric and the FMeasure metric that we defined previously.
For a measure of the general classifying performance of the expert system we use the Predictive
Accuracy metric which shows the percentage of instances in the testing set that were correctly
classified to the class they belong.
11