Accurate color conversions · Lomont.org

Accurate color conversions

Published 2023-06-06

This is a quick note for converting byte values colors in 0-255 back and forth to floating-point colors in 0-1 which avoids common errors.

Accurate color conversions

Chris Lomont, June 2023

This is a quick note for accurate conversions between color values represented as a byte value in ${0,1,…,255}$ and a floating-point value in $[0.0,1.0]$. Surprisingly, this is tricky to do well, and the most common methods found online suffer from significant problems. This result is something I’ve derived many times for gadgets (like Hypnocube stuff) over the years, and to avoid having to rederive it more times, I decided to write it out here once and for all.

TL;DR

Use the C++ code at the bottom of this post, and you’re good to go. It avoids many pitfalls for this problem you’ll find on the web.

The Problem

Abstractly, we want to convert color values represented as integers, which are how colors are generally stored in image formats, into and from floating point values, which is how colors are frequently manipulated for image processing tasks. The most common ranges are byte values representing red, green, and blue, and sometimes alpha, with the integer range ${0,1,…,255}$ to and from the floating point interval $[0.0,1.0]$. For this note I’ll write floating point values using a decimal, and integers using no decimal. Also note the integer range abstractly can be $N$ color values $0$ through $N-1$, and the analysis below still works. Here $N=256$.

We’ll start off with one requirement for color conversions, and add more as we find them.

Roundtripping : going roundtrip integer to float to integer must return the original integer value.

The bad method

The most common advice is to convert integer $i$ to float $f$ with a simple<br>$$<br>f = i/255.0<br>$$<br>This has the nice property that $0 \rightarrow 0.0$ and $255\rightarrow 1.0$ which seems right. The next question is how to convert floating values back, and this leads to trouble.

Since the first direction used a divide by $255.0$, it seems reasonable to do what most places will advise, and multiply by $255.0$ to go the other way. So we&rsquo;ll consider a first step as<br>$$<br>\hat{f} = f\times 255.0<br>$$<br>$\hat{f}$ has range $[0.0, 255.0]$. How to we convert these to integers? We can try rounding, floor, ceiling, or other methods.

Floor performs the following from half open integers (except the last, which is a single point} intervals to integers:<br>$$<br>\begin{align*}<br>[0.0,1.0) & \rightarrow 0 \<br>[1.0,2.0) & \rightarrow 1 \<br>[2.0,3.0) & \rightarrow 2 \<br>& … \<br>[254.0,255.0) & \rightarrow 254 \<br>{255.0} & \rightarrow 255 \<br>\end{align*}<br>$$<br>And right away you see the problem. Each integer $n$ can occur from a floating point range $[n,n+1)$ except the largest integer $255$, which comes from a single point. This non-uniformity is bad since it means operations on images as floating point will bias away from the value 255.

Using ceiling has the same problem, except it biases away from $0$. Rounding (we&rsquo;ll pick ties round up, any rounding mode wil have similar issues) leads to<br>$$<br>\begin{align*}<br>[0.0,0.5) & \rightarrow 0 \<br>[0.5,1.5) & \rightarrow 1 \<br>[1.5,2.5) & \rightarrow 2 \<br>& … \<br>[253.5,254.5) & \rightarrow 254 \<br>[254.5,255.0) & \rightarrow 255 \<br>\end{align*}<br>$$<br>This is better, except the integers $0$ and $255$ have half the size of interval that map to them as all the other integers. This means that image processing will lose some representation of the end colors, which is bad. This leads to the second requirement:

Uniformity : each integer should come from a uniform size floating point range (as much as possible).

We&rsquo;ll call this method the BAD METHOD (it&rsquo;s by far the most common on the web)<br>$$<br>f = \frac{i}{255.0}\<br>i = round(f\times 255.0)<br>$$

The better method

You can try lots of other methods to map, and soon you&rsquo;ll realize that the multiply by $255.0$ is the culprit. You need 256 equal sized &ldquo;bins&rdquo;, not 255. Split $[0.0,1.0)$ into $256$ bins each of width $\Delta = \frac{1.0}{256.0}$. Note for now I removed the single end value $1.0$. Similarly, consider the integers as 256 bins, each of form $[n,n+1)$. Now lets map the center in each integer bin to the center of each floating point bin. We could pick different maps, but these center maps have nice properties. This map looks like:<br>$$<br>f = \frac{i+0.5}{256.0}<br>$$<br>A reasonable inverse would be<br>$$<br>i = f\times 256.0 - 0.5<br>$$<br>and this would map exactly back, even in IEEE 754 floating point. But the float value $0.0$ and 1.0 would map outside the legal range 0,255, so we need something more careful. We want the map<br>$$<br>[n\Delta,(n+1)\Delta) \rightarrow n<br>$$<br>Multiplying the interval by 256.0 leaves a desired $[n,n+1) \rightarrow n)$, so we can use inverse<br>$$<br>i = \lfloor{f\times 256.0}\rfloor<br>$$<br>where $\lfloor t \rfloor$ is the floor function. For the single float value of...