incorrect subtraction in get color pixel value

[tilky]

for whatever reason, when i get the color pixel value of a pixel and convert it to rgb, if i subtract 20 from it, it comes out to some strange number (247 for what i was doing). I finally figured out that you need to convert the red green and blue values to singles in order to be able to do any math on the numbers and get the right answer. Does anyone have an explanation of this?

[Greg McKaskle]

I suspect that the math is being done with an unsigned 8 bit integer. In that system, 0 - 1 is equal to 255. Strange, and that is why integers are a bit of a risky tradeoff. To check this, do the math as normal, then add or subtract 256 to the number to get it back into the range of 0 to 255. That is the result that you will get with unsigned numbers.

Greg McKaskle

[tilky]

thats really weird that it would do that. do you have an explanation as to why this happens, or do you just know that it does.

[Mark McLeod]

That's normal computer science
It makes sense once you begin thinking of each data type as the bits it is capable of storing, rather than the everyday math you are accustom to.

Each data type has it's own unique data representation and we have to understand what each of them means and how it affects the values we are storing in it and how the data types affect any calculations or intermediate calculations that we may go through.

If you use a data type that can only hold 0-255, then what would you expect to happen when adding or subtracting leaves us with a number outside the possible range? If you look in the LabVIEW palettes you'll find a dozen different data types you can use. Understanding why you would use each of them is one of those skills we develop.

Integer (computer science): http://en.wikipedia.org/wiki/Integer_%28computer_science%29
Signed number representations: http://en.wikipedia.org/wiki/Signed_...epresentations

[Greg McKaskle]

Welcome to the matrix.

Since computers are typically expected to be really fast, they take lots of shortcuts. They take advantage of not doing the things that people aren't looking at, etc. They are just as capable of being careful and accurate, but that is a tradeoff, and most people prefer their video fast instead of accurate.

What you are discovering in this instance, is that it is common for images to be saved with one byte for red, green, and blue. One byte can only hold 256 unique values, so the computer scientists decided to use them as an unsigned integers. Integers are specialized and can only store the a fixed set of values. Booleans can only store two value, bytes 256, shorts can store 65536, and longs can only store a measly 4 billion values. All of these share a similar discrete mathematical system that defines the results of addition, subtraction, multiplication, and other operations.

Is the result correct? It works as intended, but not the same as a calculator. That tradeoff allows it to do specialized work quickly. Can you figure out how to get the answer you wanted? Or can you describe what answer you wanted?

Greg McKaskle

[tilky]

i figured out how to fix it by converting them to singles, which would make sure they are the same