The importance of balanced data sets for partial least squares discriminant analysis: classification problems using hyperspectral imaging data

This study investigates the effect of imbalanced spectral data in the training set, when developing partial least squares discriminant analysis (PLS-DA) classification models for use in future predictions. The experimental study was performed using a real hyperspectral short-wavelength infrared image data set collected from bakery products (buns) containing contaminants (flies) but similar applications for other insects, paper and plastic were also tested. The contaminants represent a very small proportion of the images relative to the bun. The PLS-DA model aims at accurately detecting and classifying the contaminants and this requires a modification of the calibration data set. The paper deals with problems caused by unbalanced calibration data sets and how to remedy them. In the example it was demonstrated that, by balancing the calibration data from 58,476 bun pixels + 279 fly pixels to 279 bun + 279 fly pixels, the number of true predictions could be improved with a smaller number of PLS components used in the model. The improvement for flies increased from 65% true predictions with ten PLS components to >99% true prediction with five to six PLS components. The true prediction for bun went from 100% to 99.5% with six PLS components which is an acceptable reduction. Theoretical explanations are included.