Download molegro virtual docker

13 Data Analyzer
page 200/321
Validate using external test set. The best way to validate the performance
of a regression model is to use an external dataset which has not been
involved in the training of the model. Unfortunately, the number of data
records may be too small to construct an independent set. In this case, it is
necessary to rely on cross-validation methods instead.
Watch out for chance correlation. When dealing with a large number of
descriptors and a small number of samples, there is always the possibility that
a relation between the independent variables and the dependent variable may
arise by chance. Notice that cross-validation does not automatically guard
against chance correlation: if the number of descriptors is large enough some
combinations of the descriptors will be able to describe the dependent variable.
In particular, be careful when using feature selection together with crossvalidation as model selection criteria: in this case, many combinations of
descriptors will be tested, and the combination with the best cross-validated
correlation will be found. But this correlation may have arisen by chance,
simply by trying enough combinations.
Chance correlation can be detected by validation on an external test set, but
even if this is not possible, a simple procedure exists that makes it possible to
estimate the amount of chance correlation for a dataset: y-Randomization
(sometimes called y-Scrambling) suggests that whenever a model has been
trained on a dataset, the same procedure should be applied to a dataset where
the order of the dependent variable (the target variable) has been randomized.
If the model trained on the randomized dataset yields a high cross-validated
accuracy, the correlation is caused by chance. Notice that it is important to
start the model building from scratch on the randomized dataset: if feature
selection was performed on an initial set of descriptors, perform the feature
selection once again on the randomized dataset – do not try to build a model
on the randomized set with the descriptors chosen from the initial dataset. It is
the whole procedure that must be repeated in order to estimate the chance
correlation. Also notice that the randomization and model building should be
repeated a number of times in order to get an estimate of the magnitude of
the chance correlation. Y-Randomization can be performed by choosing
Preparation | Scramble Selected Columns (see Section 13.18).
Check for obvious outliers. It may be difficult to decide whether abnormal
data are outliers or not – and it may be scientifically questionable to remove
them. However, the datasets should always be checked for obvious data errors
arising from e.g. preparation or conversion faults.
molegro virtual docker – user manual