sethmlarson.dev

When str.lower() is a security vulnerability in Python — Seth Larson

much more than the Latin alphabet. Thus, a mapping from Unicode to

(IDNA), also known as “IDNA 2003”. The StringPrep algorithm is defined in

Python supports IDNA 2003 through the idna codec ( str.encode('idna') ) and

IDNA 2008 is supported by the idna package on the Python package Index .

in the standard library. In general, you should be using the idna package (IDNA 2008)

StringPrep defines the “case folding” step (case folding is approximately “how to lowercase/uppercase a codepoint”) in Section 3.2 , enabling case-insensitive

comparisons of strings, by mapping all characters through mapping tables B.2 and B.3 .

B.2 is effectively str.lower() , lowercasing all characters according to

Unicode rules and B.3 contains the exceptions. The Python code implementing

this (and assuming B.3 table is captured correctly) is the following code below:

And that might seem fine... and the title probably gave it away already.

Why? Because str uses whatever Unicode data that the particular Python

There's also a database of Unicode 3.2.0 data available on every version

of Python ( unicodedata.ucd_3_2_0 ) specifically for the StringPrep and IDNA algorithms:

This is important! StringPrep depends on this specific version of Unicode

to operate consistently, the B.2 and B.3 tables in RFC 3454 are essentially