handle Unicode uppercase/lowercase conversions correctly

[StefanKarpinski]

As pointed out here, there are cases where a single Unicode character needs to be split into multiple characters when converted to uppercase: e.g. ß to SS and ffl to FFL (it's possible that similar cases exist for conversion from uppercase to lowercase as well). The interface of the towupper and towlower functions which we use for general Unicode case conversion can't handle such transformations since the signature is Char to Char. Despite this, we should handle these conversions correctly, although I'm not sure where to get code that does this.

[StefanKarpinski]

More information can be found here: https://www.unicode.org/faq/casemap_charprop.html. On a related note, we should handle correct titlecasing of strings as well, which has its own difficulties.

[6e441f9c]

It looks like the main drawback of ICU is that it converts data to UTF-16 internally by default; there are some UTF-8 friendly interfaces, though

[StefanKarpinski]

What is ICU?

[nolta]

IBM's Unicode library: https://site.icu-project.org/

[StefanKarpinski]

We can maybe just steal the code for case mapping, assuming they handle this issue correctly. We already handle almost all Unicode issues just fine. This is just one that even the standard C Unicode functions fail on.

[JeffBezanson]

I'm not even sure what changing letter case is for. We have towupper just because it's there. Case folding or case-insensitive searching seems slightly more useful. One question is whether we want to have our own unicode tables, which can be pretty massive. People do all kinds of tricks generating code from them etc.

[StefanKarpinski]

I'm not even sure what changing letter case is for.

Really? You are clearly not a Perl programmer :-P

[nolta]

It looks like the main drawback of ICU is that it converts data to UTF-16 internally by default; there are some UTF-8 friendly interfaces, though

Fortunately, these include the case mapping functions. I've wrapped them up in extras/icu.jl:

julia> uppercase("testingß")
"TESTINGß"

julia> load("icu.jl"); import ICU.*;

julia> uppercase("testingß")
"TESTINGSS"

[aviks]

I think the ICU package works well for this. Do we want to have the unicode tables in base at any point? Should this issue be closed?

[JeffBezanson]

Are we planning to add this to Base?

[StefanKarpinski]

It seems like a lot of stuff to put into Base. The ICU library is kind of huge for just this one tiny and not-very-common piece of functionality. It would be nice to be able to pull the correct uppercasing logic out of ICU – or get it from somewhere else.

[JeffBezanson]

Even just that is quite big --- you need the full unicode tables.

[StefanKarpinski]

Then I'd say the current situation is a local optimum: we have mostly correct basic uppercasing and lowercasing in Base and if you need the full fancy version, then you use ICU and get it.

[aviks]

+1

[hayd]

Did titlecase drop off the radar here? (I thought you could do it in the past, but can't find any related issues)

[nalimilan]

@hayd At least that's supported by https://github.com/nolta/UnicodeExtras.jl.

[stevengj]

Titlecase info is provided by UTF8proc, but it would be nice to have a little wrapper routine like utf8proc_uppercase to make it easier to access.

[stevengj]

Note that utf8proc 2.0 added utf8proc_totitle, so it would now be trivial to add a titlecase function in Julia.