'\xc0\x80' should either error or make an overlong Char

[StefanKarpinski]

Currently we have this:

julia> '\xc0\x80'
'\0': ASCII/Unicode U+0000 (category Cc: Other, control)

julia> '\xc0\x80' == '\0'
true

julia> String(b"\xc0\x80")[1] == '\0'
false

In other words, you can write an overlong Char literal as bytes and you get the standard character representing that code point. This should either be an error or, preferably IMO, now that we actually have Char values for overlong encodings of characters, it should produce that Char value (i.e. the one I've produced via byte string literal syntax). This should technically be done before the 1.0 feature freeze, but I really doubt anyone is relying on this, so I think we can sneak it in whenever.

[StefanKarpinski]

Jeff, what's the best approach to fix this? The complication as I understand it is that we use femtolisp chars to represent char literals when parsing and lowering and femtolisp can't represent invalid chars like Julia can. Behaviorally, I think that if "<...>" creates a one-character string, even if that character is invalid, we should have '<...>' == "<...>"[1]. This violates that since we have this discrepancy:

julia> '\xc0\x80'
'\0': ASCII/Unicode U+0000 (category Cc: Other, control)

julia> "\xc0\x80"[1]
'\xc0\x80': [overlong] ASCII/Unicode U+0000 (category Cc: Other, control)

What I'm proposing is that '\xc0\x80' should produce the same overlong Char as "\xc0\x80"[1]. Similarly, I think that these invalid char literals should be allowed to work:

julia> '\x80'
ERROR: syntax: invalid character literal

julia> '\xff'
ERROR: syntax: malformed expression

(I'm not sure why these give different errors, they probably shouldn't.) This change would bring them into line with the corresponding invalid string syntax:

julia> "\x80"[1]
'\x80': Malformed UTF-8 (category Ma: Malformed, bad data)

julia> "\xff"[1]
'\xff': Malformed UTF-8 (category Ma: Malformed, bad data)

I believe such a change would bring char and string literals into full syntactic agreement.

[JeffBezanson]

I think the most direct way to do it is to replace the call (string.char str 0) near the end of the code for char literals in parse-atom with a call that behaves like julia getindex. At that point, str is exactly what you'd get if you wrote double quotes instead of single quotes. Flisp aref can be used to get bytes from a string.