throw error if a bare string literal is invalid UTF-8

[StefanKarpinski]

See this thread. This changes the behavior for bare string literals previously described in #4 to be the following:

• all bytes < 0x80: ASCIIString
• has bytes ≥ 0x80 and is valid UTF-8: UTF8String
• invalid UTF-8: throws an error

The b"..." string form (see #11) will let you use string syntax with \x and \u to make byte arrays. If you want to make a UTF-8 string that contains invalid UTF-8, you can do something this:

UTF8String(b"\xff\xff")

[ViralBShah]

Seems like this can be in 2.0, unless it is easy enough to do. Stefan?

[StefanKarpinski]

Yeah, it turns out to be a pain in the ass to do from the C code (can't find the email thread, but it is).

Another possibly simpler way to handle this is to change the parser to disallow \x80 through \xff in bare string literals but to allow them in prefixed string literals, passing them through to implementing macro to decide whether to allow them or not. Throwing a syntax error from a macro is no problem, it's throwing a syntax error from C code called directly by the parser that's problematic. @JeffBezanson, can you do that?

[JeffBezanson]

Actually it makes sense anyway for the parser to give a syntax error on invalid utf-8 sequences in a string or anywhere else. The parser can also check in cases where it does unescaping, and the julia string macros can check when they do unescaping. In case of an error the macro should return an error expression, Expr(:error,{"msg"}) instead of throwing, to allow history to work properly.

Also this seems strange:

julia> "\\\""
"\\\""

julia> S"\\\""
"\""

Am I doing something wrong in the parser?

[JeffBezanson]

Also I see utf32.j is not in use. Can we get it in shape?

[StefanKarpinski]

I could work on utf32.j and latin1.j or we could mothball them for the release. The main issue is that currently we don't actually have any support for reading files with different encodings. These aren't really useful without that.

There's definitely something funny going on with escape handling that's different for bare vs. prefixed or otherwise macro-handled strings. See issue #100. Pretty sure it's the same problem. I'm looking into it, but I'm not quite sure what's going wrong yet.

[JeffBezanson]

Ah, I was wrong, a macro can throw an error and it is automatically handled.

[StefanKarpinski]

How does 6614948 not address all of this issue? Seems fully addressed to me.

[JeffBezanson]

Oh, I guess that's true since print_unescaped never generates invalid sequences. But we have this:

julia> "\x80"
syntax error: invalid utf-8 sequence

julia> "\x80$1"
"\u00801"

We have to do something else with \x and \000. In byte arrays, b"\x80", we clearly want \x to insert bytes. So to be consistent it should always insert bytes, and for strings this process is followed by a check to make sure all those bytes add up to valid utf-8.

[StefanKarpinski]

Ah, yeah. That is still an issue. I'll add a check after constructing a new string. If we just disallowed the escapes above \x7f altogether checking UTF-8 validity would be unnecessary since there'd be no way to even express an invalid string. That would have to match between the parser and the str_S form though — shouldn't allow it in one and not the other.

[JeffBezanson]

Yes that would also be a sensible approach. It's kind of a toss up. I prefer to err on the side of allowing as much as possible. You can enter anything, but we call a validation routine. It's "trust but verify" :)
The validation is needed anyway since the source file itself could contain invalid utf-8 even with no escape sequences present.

Plus, after fixing this we get byte array literals for free.

[StefanKarpinski]

Closed by ad06687.