We handle this in a few ways
First off, the data we collected is from a variety of positions and lighting conditions. That includes weird angles, in and out of shadows from the sensors (and students) and so on. That variety of training data should help the network handle a range of inputs and correlate it with the correct sensed color.
The other thing which helps is that the output of the net is a confidence value for each color. 0 meaning perfectly confident it is not a given output color, and 1 being confident it is. We can use thresholds on this confidence to figure out what to do. If the confidence is high for a single output and basically 0 for the rest, we can be comfortable that the net is predicting correctly. If the confidence is low for all colors, or perhaps moderate for a few, we can use that to mean we don’t have a good detection.
Another option which we didn’t explore yet is to collect data when the sensor is pointing at random non-color-wheel colors, and then marking that data with all the output classes set to 0. This would perhaps help train the net to understand what the sensed colors aren’t, if that makes sense. Care would need to be taken to not overwhelm the training data with negative examples like this, but it might help in some cases.
Hopefully that helps?