Bernhard R. Links' blog -- Who made this description field required?
Bernhard R. Link's blog
Welcome to my little private blog.<br>Most of what you find here is targeted towards planet.debian.org,<br>so it might make sense to follow that to understand what I am talking about.
The contents of this blog are of course also available as rss feed, even<br>as two feeds:
index.rss<br>one with all posts<br>changelog.rss<br>only the posts in (an attempt of) English and that are targeted to planet.debian.org.
For more information who I am and how to contact me, take a look at<br>my website (German only).
I learned something new about URLs today
Today I stumbled over some behavior that I found quite surprising:
$ ipython3 -c 'import httpx;print(httpx.URL("https://example.com/foo/bar/../../baz"))'<br>https://example.com/baz
Even more surprising that behavior is actually standards-compliant,<br>even mandated by RFC 3986.
The underlying motivation is relative reverences.<br>If some resource reachable by "https://example.com/foo/bar"<br>references another resource relatively as "../../baz" then<br>this is of course the intended result.
Getting from this problem to what RFC 3986 suggests<br>might be surprising in the result, but somewhat understandable if you<br>look at the consequences of that problem:
Giving the path components ".." (and ".")<br>special meaning at the start of the relative reference means that if you allowed<br>them in absolute URLs those would be impossible (or at least very convoluted)<br>to address as relative URLs.
So RFC 3986 describes a way to handle them everywhere:<br>Just join the path of the base URL and the path of the relative reference<br>and normalize the result. Or normalize the absolute on either side if only<br>that is to be taken.<br>This makes things very convenient:<br>Multiple reference URLs can just be joined without special handling for<br>relative references starting with dots, making writing applications handling<br>them easier.<br>Programmers don't have to care how to handle relative references and can<br>just join everything in whatever way they want.
For maximum elegance there is still some corner case left:<br>What happens if an absolute URL has a path starting with double-dot components?<br>Or an relative path starting with more of them then the base URL's path has components.<br>You just ignore them:
$ ipython3 -c 'import httpx;print(httpx.URL("https://example.com/../../baz"))'<br>https://example.com/baz
With that last point every URL is valid and has well-defined meaning.<br>Handling relative references and relative paths is very easy and convenient.
So this shows a high regard for simplicity, elegance and convenience.<br>And a total and uncompromising disregard of security.
After all the most convenient it is for an attacker;<br>If they are allowed to supply a path component for a request a system<br>does in their behalf, then they can easily escape anything they were supposed<br>to be limited to.<br>The ignoring of dots at the start means they don't even have to know<br>exactly how deep their request is:
$ python3 -c 'import httpx;print(httpx.URL("https://example.com/public/api/public/resources/harmless/../../../../../../../../../internal/data"))'<br>https://example.com/internal/data
So even if the resource server securely handles request<br>(unless you consider not having any way to lower your permissions for one request to a specific subset),<br>your fully RFC conforming client library will already request the permission they should not have permission for.<br>Even worse dots are usually not characters you can easily forbid so once slashes are to be allowed things get complicated.
There also would have been a simple, elegant and secure way:<br>Consider every path element ".." or "."<br>in an (absolute) URL an error.<br>Define a reference resolution that allows the relative reference to only start<br>with "./" or one or multiple "../" and<br>consider every appearance of a dot or two dots as path components after than an error.
Everything joining two paths has to either use an implementation of that path joining<br>algorithm, but only if they want to joins paths in the potentially dangerous<br>way allowing leading "../". Otherwise they can just use the normal<br>join and even if an attacker gets those dots that will just cause the generated URL<br>to be rejected as invalid.
Of course using a secure implementation is now even more inconvenient thanks to RFC 3986 being around:<br>If you have no control over the generator of relative references, it is always possible<br>that they generate relative references with ".." components after non-dot<br>components.
And if you check all code to properly filter out "/../",<br>keep in mind that convienence does not stop there.<br>After all it is not unheared of for server implementations to helpfully normalize<br>unicode characters, too, or translate them to their nearest ASCII equivalents.<br>Or translate percent escaped characters back before doing path splitting.<br>Or you might think there was some unicode codepoints between those two dots,<br>but they that those were some...