Download ACRES 3 User Guide
Transcript
recall = 2 × precision × recall TP TP F _ measure = precision = precision + recall TP + FP , TP + FN , Finally the weighted average of these metrics for all classes can be calculated, taking into account the number of occurrences of each class in the dataset. These metrics are widely used in classification performance evaluations and corresponding tools like the data mining tool Weka so using them allows the direct comparison with various classification models. Evaluation Report in ACRES The dataset is partitioned in two sets: training and testing set. The expert system is generated using the training set and then it is evaluated on the data of the testing set. The procedure is repeated for different partitions of the dataset (cross validation) and the average values of the metrics are presented. The produced expert system does not simply classify an instance to the predicted class. It provides an uncertainty value for each possible class. In order to make the evaluation of a produced expert system easier, we consider that the system classifies an instance to the class for which the uncertainty factor is the highest. As described above, to evaluate an expert system for more than two classes, we evaluate the performance for each class separately. For each class i, we treat the problem as binary, with the first class being i and the second class being a class consisting of all other classes. We then form a confusion matrix and compute the metrics for each class i. We are mostly interested in the Sensitivity and Precision metrics. We also combine these two metrics producing their mean proportional, SQRT(p*r) as a more general metric and the FMeasure metric that we defined previously. For a measure of the general classifying performance of the expert system we use the Predictive Accuracy metric which shows the percentage of instances in the testing set that were correctly classified to the class they belong. 11