Should you normalize RGB values by 255 or 256?

30fps.net — Computer Graphics & Programming with Pekka Väänänen

Let’s say you’re writing an image processing program.<br>The program takes in an image, converts it to floating point, does some processing and finally saves the modified pixels to disk as 8-bit colors.<br>The question today concerns how exactly the integer-to-float conversion should be done.<br>There are two approaches which, written in Python and NumPy, look like this:

Standard division by 255

Alternative division by 256

pixels = img / 255.0<br>result = process(pixels)<br>output = np.trunc(result * 255 + 0.5)

pixels = (img + 0.5) / 256.0<br>result = process(pixels)<br>output = np.trunc(result * 256)

I assume that in both cases the output values are clamped before the final typecast:

# Clamp and cast to 8 bits<br>output_8bit = output.clip(0, 255).astype(np.uint8)

The standard approach maps the integer 0 to 0.0 and 255 to 1.0.<br>It works perfectly fine and is how GPUs do it.<br>The alternative adds a 0.5 bias and divides by 256 instead, so the integer 0 gets mapped to 0.5/256=0.001953125.<br>This is inconvenient because your image processing code can’t detect black pixels, for example, without knowing the above constant.<br>As a consequence, you tie your logic to 8-bit inputs even if you compute in floating point.<br>With the standard approach, you can always assume black is 0.0.

But some programmers still feel a pull towards the alternative.<br>What is going on?<br>What do they see in it?

The case against 255.0

The standard approach does look quite strange when plotted on the number line.<br>Below you can see an exaggerated version with 3-bit integers in the range [0..7] being mapped to [0,1]:

On the X-axis we’ve got a number line and the locations of brown circles on it represent the decoded floating-point values.<br>The numbers inside are the integer inputs.<br>Each integer has arrows pointing to it; these show a range of floating-point values that round to it.<br>I’ll call these ranges “bins” in the rest of this article.

Smaller bins at the extremes

The first issue really apparent in the diagram is how the standard formula’s extreme bins jut beyond the [0,1] range.<br>Perhaps this visualization is unfair – both approaches clamp their output so the extreme bins could extend infinitely – but it clearly shows how “stretched” the standard range is.<br>The stretched range is wider than the assumed operating range [0, 1] in image processing.

This means that when converting floating-point values in the [0, 1] range back to integers, the extreme bins have effectively half the width of other bins.<br>As a consequence, it will be “harder” to output extreme values from your algorithm.<br>For example, if you generate uniform [0,1] noise and round it using the standard formula, the values 0 and 255 will occur only half as frequently as other integers.

We can verify this claim empirically by generating a million uniform random numbers, plotting them as a histogram, and observing that both the 0 and 255 bins are indeed only half as tall as other bins:

The highlighted crop:

Histogram code

import numpy as np<br>import matplotlib.pyplot as plt

result = np.random.uniform(0, 1, 1000000)<br>final_values = np.trunc(result * 255 + 0.5).clip(0, 255).astype(np.uint8)<br>plt.hist(final_values, bins=256, range=(0, 255))<br>plt.show()

Still, I’m having a hard time coming up with an example situation where the bias away from the extremes would prove problematic.<br>Sure, the standard approach’s floats are spread over a wider range, but the original image will still round-trip convert losslessly (uint8 → float → uint8).

Also, any result value just beyond 0.0 or 1.0 will still round to the right bin, evening out the output distribution.<br>An example of what I mean.<br>Assume your processing subtracts 0.005 from the floating-point colors.<br>In the standard approach this pushes blacks below zero – outside the [0,1] range – but in the alternative the values stay positive.<br>In the end both output the integer 0 anyway:

Standard:<br>trunc(255 * (-0.005) + 0.5) = 0

Alternative:<br>trunc(256 * (0.5 / 256 - 0.005)) = 0<br>It didn’t matter that in the standard approach the zero bin was only “half the size”.

Inexactness

The second issue is that the standard approach’s floating-point values aren’t exact.<br>For example 128/255.0 \approx 0.501961 but 128/256.0 = 0.5.<br>Due to this round-off error, the distances between floating-point values vary a tiny bit.<br>But this isn’t a real problem since the error is truly tiny.<br>A 32-bit floating-point number has a 23-bit fraction (“significand”).<br>We are talking about round-off error in its least-significant bit; jitter with the magnitude less than 2^{-23}.<br>Surely a relative error of 0.00001 % is immaterial even in the most sophisticated image processing task.<br>In this case, inexactness is an aesthetic question, not a technical one.

Values not in between integers

The alternative approach always places each floating-point value exactly in the middle of two integers.<br>See how...