When str.lower() is a security vulnerability in Python – Seth Larson (sethmlarson.dev)
89 points by rbanffy 8 hours ago
echoangle 7 hours ago
> This is why calling str.lower() represents a difference in the implementation and the specification, and therefore a vulnerability:
I wish there was some explanation how this is a vulnerability and not just a bug generating erroneous data.
Vulnerability for me sounds like there’s a reasonable way to create an exploit from the bug, and I don’t see one here as someone who’s not very familiar with the topic.
tptacek 7 hours ago
It creates a parser differential; two different components of the system can treat the same string as different hostnames. Things that have trusted hostnames, or privileged/admin hostnames that are screened out, or SSRF filters all depend on accurately comparing presented hostnames.
This is pretty situational, though, isn't it? You still have to be dealing with IDN names.
tialaramex 6 hours ago
It is situational, but it very much seems like a thing you'd squirrel away and bring out when you find a system where the differential is helpful.
DNS names are a thing where Sales is going to tell the Engineer that they can't issue the customers randomized ASCII names like abxuewrf.my-thing.example because real customers want to write our-brand-name.my-thing.example instead - even though you already know bad guys will choose billing.my-thing.example and name-of-bank.my-thing.example and every other unintended bad choice even before we realise about likelihood of these confusion bugs in software like Python.
xp84 6 hours ago
inigyou 43 minutes ago
TZubiri 2 hours ago
I can see how this would be a vulnerability in the context of a security researcher that wants to exaggerate their findings in order to get paid a bounty.
SethMLarson 7 hours ago
Author here, that's a good idea. A straightforward way to exploit an implementation differential like this is if you have a software system that contains two different implementations of IDNA 2003 processing user input. One part of the process processes the domain correctly, the other incorrectly, and in this case you can have one part of a system (such as a policy/filter) "see" the data one way and the other part of the system (such as, taking an action as a result of the data) see the data in another way.
Server-side Request Forgery (SSRF) is an example of such an exploit targeting a differential in implementations of URL parsers, which is similar to this implementation difference.
reichstein 36 minutes ago
I guess that's why you shouldn't be _validating_ things, and then praying them on as-is. "Parsing is validation" is (also) the idea that after validation, your data is in a format that reflects and enforces what's known about it.
"Safe strings" is an example of that idea. Not always possible or practical, but always worth considering if you're doing "validation" as a function.
jerf 3 hours ago
I wouldn't call this a "vulnerability", I'd call it "a thing that can potentially turn into a vulnerability, more often it can turn into an obscure bug, and most often it is just a quirk".
In particular, if my corporate security team started just mass-flagging all instances of "str.lower" as "security bugs" I would be having a talk with their manager about their threshold for what constitutes a "security bug". Their job is arguably to be more sensitive to that than most engineers, but not that sensitive. It would be like flagging all instances of string concatenation as a vulnerability... and I say that as the guy who would like to eliminate simple string concatenation from programming languages, already a very extreme position on that operation, because of it being at the root cause of so many vulnerabilities... but simply flagging every use as a "vulnerability" is way too sensitive. A demonstration of the ability to use it to bypass some sort of security barrier is necessary to call any specific instance a "vulnerability".
And string concatenation has caused orders of magnitude more actual, verified vulnerabilities than incorrect case folding has.
catlifeonmars an hour ago
gcr 4 hours ago
That would be a vulnerability in the IDNA filter that they’re responsible for fixing.
AgentOrange1234 7 hours ago
"if you have a software system that contains two different implementations of IDNA 2003 processing user input"
Is that a real thing though? Is someone doing that?
kccqzy 7 hours ago
rcxdude 6 hours ago
cwillu 6 hours ago
gchamonlive 7 hours ago
floxy 7 hours ago
I suppose I could envision a scenario like: Service A has a "reset your password" option. Someone with a "user@popular-unicode-domain.com" tries to use this option. Normally this fails when mangled-popular-unicode-domain-plus-garbage.com can't be found. Enterprising malicious actor registers mangled-popular-unicode-domain-plus-garbage.com, and now gets a hold of user password reset requests.
wren6991 6 hours ago
My favourite example of this is the Chromium bug where enabling floating point flush-to-zero for WebAudio was used to cause deliberate heap corruption: https://issues.chromium.org/issues/382005099
> We have a working exploit (OOB access in the V8 heap), our security folks put one together based on the example I posted above (and they're cleaning it up to post it here). In general, we find that correctness issues like this are pretty much always exploitable with a bit of effort (not even that much effort normally, just gluing together a few gadgets), so we treat correctness issues as security issues until they are proven not to be, rather than the other way around.
The floating-point-to-heap-corruption chain here is... uniquely JavaScript, but in general getting two different implementations to disagree is the start of lots of interesting inconsistent behaviour.
zaphar 7 hours ago
I'm too lazy to investigate further but my guess is that if there is a vulnerability here it has something to do with dns name spoofing.
tialaramex 7 hours ago
This idiocy is a big part of why it was so important to get Python people working on TLS implementations to understand that the defined mechanism for SANs (no the "alternative" in Subject Alternative Name doesn't mean in the sense of more than one, X.509 is originally for the X.500 system and the Internet repurposed X.509 so these are alternative names from the Internet) says that these are DNS names, they specifically are not to be understood as some sort of human readable text, and thus "decoding" them to Unicode is definitely nonsense even though Python really wanted to do that and I think used to do it or at least proposed to.
The rule for how SAN DnsNames match againt like names, from the DNS is very, very simple so that you don't screw it up. You handle a single wildcard (ASCII * code 42 matches any single DNS label) and beyond that it's literally byte comparison. You don't care what these bytes mean, either the bytes are all identical or that's not a match and we're done.
ummonk 4 hours ago
> The fix was to create new exceptions so that str.lower() would behave as if it was using Unicode 3.2.0 for only particular function. So, we go through each Unicode codepoint and record when the behavior of str.lower() is different when comparing the Unicode version shipped with Python and Unicode 3.2.0
This sounds like a really hacky solution compared to implementing a separate frozen Unicode 3.2.0 lower.
jooon 6 hours ago
Reminds me of an old security incident at Spotify https://engineering.atspotify.com/2013/06/creative-usernames
ajd555 4 hours ago
So the attack surface would be a bit flip on a domain name? Or more specifically, a unicode conversion flip, where an attacker could redirect to a malicious IP?
Impressive to have found such a vulnerability!
inigyou 44 minutes ago
If this is so important to know you probably shouldn't serve 403 errors to people.
K0IN 6 hours ago
I was also startled when python did ß.upper() returns "SS". Which is kind of unsuspected in some cases (if string length changes with an upper call)
smallerize 4 hours ago
That's in the standard. https://www.unicode.org/reports/tr21/tr21-5.html
[SpecialCasing] Contains additional case mappings that map to more than one character, such as "ß" to "SS".
kevin_thibedeau 2 hours ago
5.1 adds uppercase ẞ which can fold to either ss or lowercase ß depending on the chosen algorithm.
ike_sh 2 hours ago
Hit this with the Kelvin sign once. Took embarrassingly long to track down.
cat-whisperer 7 hours ago
the horrors of versioning
drdexebtjl 3 hours ago
All 6 users of IDNs must be really worried right now.
tescreal 7 hours ago
it would be good to look for ".lower()" squatters if you host on an idn.
ittsel 5 hours ago
if you have a software system that contains two different implementations of IDNA 2003 processing user input" Is that a real thing though? Is someone doing that?
ittsel 5 hours ago
Is that a real thing though?
bawolff 5 hours ago
That's a little over the top. string.lower() is not a security vulnerability. Not following the spec is the security vulnerability.