utf8proc doesn't support unicode 6

[johnmyleswhite]

When I try to produce a subscript t on the letter "a" by typing in a\_t<TAB>, I get the following behavior:

julia> aₜ
ERROR: syntax: invalid character "ₜ"

In contrast, I can safely produce aᵣ.

[JeffBezanson]

This is due to utf8proc not supporting unicode 6.0.0. See also #6939

[stevengj]

Unfortunately, I haven't been able to reach the utf8proc maintainer since February. When I contacted him at the time, he said he was willing to make a public Mercurial repo for utf8proc, and was interested in updating for Unicode 6, once he had done some cleanup.

We may have to make a fork of utf8proc in order to import the Unicode 6 tables and make other changes that would be useful for us (e.g. to import the Unicode character-width tables for #6939).

(We could also switch to something like ICU, but I'm a bit reluctant to go that route. utf8proc is a clean, small C library based on UTF-8 that does what we need and no more, while ICU is a huge C++ library based on UTF-16, with lots of extraneous functionality.)

[StefanKarpinski]

I agree. Let's fork utf8proc. Maybe rename the project to avoid confusion? Calling it libutf8 might be good. We would of course make the derivation from utf8proc clear.

[stevengj]

Maybe libutf8proc, just to make the relationship clearer in the name.

[StefanKarpinski]

Seems reasonable.

[stevengj]

I have a cleaned-up git repo as a starting point for work on utf8proc. Should I start a JuliaLang libutf8proc repo?

[jiahao]

go for it

[StefanKarpinski]

yep, that sounds good.

[stevengj]

I don't have admin rights to JuliaLang, so I can't create a repo there. I temporarily put it at stevengj/libutf8proc and added @StefanKarpinski and @jiahao as collaborators. Either one of you should feel free to transfer the repo to JuliaLang.

[stevengj]

Or, no, it looks like you have to give me temporary admin rights to transfer it in.

[StefanKarpinski]

I gave you admin rights. You can now transfer it or create it.

[stevengj]

Okay, moved to JuliaLang/libutf8proc JuliaLang/libmojibake

[abcsds]

Hi I reproduced this problem on Julia 0.5.0 Should I post it to JuliaLang/libmojibake ?
I'm using latex symbol \mid
Thank you

julia> ∣0⟩ = [0,1]
ERROR: syntax: invalid character "∣"

julia> versioninfo()
Julia Version 0.5.0
Commit 3c9d753 (2016-09-19 18:14 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM)2 Duo CPU     P7550  @ 2.26GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Penryn)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, penryn)

[yuyichao]

1. It's not libmojibake anymore but back to libutf8proc
2. Please don't post in an old closed issue. It's most likely unrelated to your problem
3. IIUC, we intentionally don't allow certain symbols to be used in variable names since they are too confusing with other ones.