After a well-earned vacation, I would like to sort my photos automatically into two classes: those taken on the beach and all the others. We can imagine a simple solution to this problem. Given an input image, I can count its yellow pixels. In the RGB system, where a pixel’s color is characterized by the amounts of red, green, and blue it contains, we might test whether the red and green values are "high enough" and the blue value "low enough." The hope is that images containing enough yellow pixels will be the beach photos.
How should we define "enough"? We choose a threshold: if the number of yellow pixels in an image exceeds it, we classify the image as "beach"; otherwise, we classify it as "other." This defines a decision function. If the threshold is too high, all the files that exceed it will indeed be beach photos (because they are "very yellow"), but we may wrongly reject beach images that contain relatively little sand and therefore do not have enough yellow to exceed the threshold. Conversely, if the threshold is too low, we run into the opposite problem: most images will be classified as "beach," including photos that actually belong to the "other" class.
-
A threshold and penalties -------------------------
Instead of choosing such a threshold arbitrarily, we will create a training set by manually classifying around fifty of these photos into two categories ("beach" and "other"). With this training set in hand, we can now associate an error rate with each threshold. Given a threshold, we count the yellow pixels in each image in our training set and compare the result with the threshold. If the classification is correct, there is no penalty; otherwise, we incur a penalty (normalized to 1, say). The "best" threshold for this training set is therefore the one that gives us the lowest average penalty. If the photos in the training set are representative of the entire album, we can expect performance on the training set to reflect actual performance on the album as a whole; this is known as generalization.