Proposal: Unicode source files

[zygoloid]

Derived from #17.

• RFC topic
• Decision topic
• Decision announcement

[jonmeow]

Generally seems fine. I think my only real suggestion is that you might consider moving the bulk of content (background, details, alternatives, etc) to something like docs/design/lexical_conventions/unicode_files.md. This suggestion is mainly reflecting a perspective that we should aim to keep alternatives with the design where it's easy to find, and using the proposal as suggested here may make it harder to track. Plus, then you can incrementally build the structure of lexical conventions back in that way.

[zygoloid]

I think my only real suggestion is that you might consider moving the bulk of content (background, details, alternatives, etc) to something like docs/design/lexical_conventions/unicode_files.md.

Thanks, I think this is the right place for this content to end up, and I agree that we should include rationale and alternatives and such there. For this proposal I'd like to get a decision before I start working on docs/ updates, if that's OK?

[chandlerc]

Like the direction, most of this is about cleaning up the proposal a bit for posterity.

[chandlerc]

Somewhat meta comment here and elsewhere:

While I absolutely agree on aligning this with the relevant parts of the Unicode specification, I'd like to make the proposal and especially any eventual design document built around this stand alone.

Could you rotate the proposal a bit to outline the functional constraints and then identify how they map into the Unicode standard? Some of these I'm guessing are large (annex 31 here) and in those cases maybe just provide illustrative examples of familiar subsets of the rules to help readers understand the overall direction w/o needing to dig up the annex in question.

[chandlerc]

Just to clarify this a bit: I'm not asking for lots of detail or anything long here. I like the short and directionaly focused proposal, and don't want to turn it into a spec or anything else. I think it's totally fine to defer to when you're pulling the full design docs together and/or touching the spec for that, and deferring to the Unicode docs here.

I just would like to be able to get a high level rough understanding of the consequences of this direction w/o the indirection.

Feel free to prod me on chat if this is feeling like a bunch of work. =]

[zygoloid]

I've expanded this section and the section on normalization to explain a bit more about the contents and consequences of the annexes in question.

[mconst]

Do we actually want to use UAX#31 as our baseline here? My feeling is that it's far too permissive, which makes security very difficult (#19) and also adds unnecessary complexity (e.g. the whole discussion in #17 about allowed whitespace characters, when all we actually want are spaces, newlines, and possibly tabs).

It would be great if we could find a preexisting design that does what we want, and I do think we should adopt any parts of UAX#31 that are useful, but I think any design that solves the security issues is going to end up looking pretty different overall. My guess is it makes more sense to start with something restrictive but secure, and add support for more languages over time, rather than starting with something permissive and trying to patch up the security holes.

[zygoloid]

One of the overarching unanswered questions in #19 is to what extent we want to mitigate the risk of intentionally-underhanded code -- do we want to defend against only Murphy or also Machiavelli?

To some extent I want to defer to UAX#31 for now because, with an essentially unbounded set of places in which we can explore and innovate, this seems like neither a place where we have a lot of expertise nor a place where breaking new ground is going to give the best impact in terms of meeting our goals -- I think this is more an area where we want a reasonably-standard and uncontroversial answer.

[chandlerc]

(I just want to say my original comment has been addressed... didn't really want to resolve the thread because of @mconst 's comments)

[gribozavr]

The cost will need to be paid by a compiler that cares about good error messages -- we would need to accept non-NFC identifiers, produce an error, but then normalize them to NFC, and ensure that name lookup works. Maybe we can make NFC the fast path (http:https://unicode.org/reports/tr15/#NFC_QC_Optimization), but a good compiler would still need to normalize in erroneous cases.

[zygoloid]

That's a good point. I'll add that to the document, but do you think we should actually reverse the decision and allow arbitrary un-normalized text?

[gribozavr]

I was only pointing out that this disadvantage might not be material in practice.

We could add another disadvantage if we want, explicitly calling out naive tools that process bytes. Do grep or git grep care about normalization differences for example?

[mconst]

Neither grep nor git grep do normalization themselves, so they do care. (Note that they're not naive byte-based tools -- they both support Unicode case-insensitivity, which is pretty complex! They just don't do normalization, which isn't a huge problem in practice because most real-world text is in NFC already.)

[chandlerc]

Note that I think the complexity is still significantly reduced by requiring normalization. That means the compiler can take an approach such as a fallback path that renormalizes and re-starts lexing (or other potentially costly fallback strategy) rather than needing to normalize in the "expected" path.

But perhaps the greatest reason to require it is to get diff stability and byte-equivalence for things like string literals.

[zygoloid]

The document was revised to incorporate @gribozavr's comments, but to require normalized text everywhere based in part on @chandlerc's observations. I consider this thread resolved.

[josh11b]

On Mon, Aug 24, 2020 at 11:48 PM Richard Smith ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In proposals/p0142.md <#142 (comment)> : > +### Normalization + +Carbon source files, outside comments and string literals, are required to be in +Unicode Normalization Form C ("NFC"). The Carbon source formatting tool will +convert source files to NFC as necessary to satisfy this constraint. + +### Characters in identifiers and whitespace + +We will largely follow Unicode Annex 31 in our selection of identifier and +whitespace characters. This Annex does not provide specific rules on lexical +syntax, instead providing a framework that permits a selection of choices of +concrete rules. One of the overarching unanswered questions in #19 <#19> is to what extent we want to mitigate the risk of intentionally-underhanded code -- do we want to defend against only Murphy or also Machiavelli?

I would definitely be in favor of selecting a less permissive, more secure option. Do we have any good choices here?

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <#142 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADUNHV5CDFOXU7HZOMCDORDSCNNFFANCNFSM4P7AUUOQ> .

[zygoloid]

@josh11b

I would definitely be in favor of selecting a less permissive, more secure option. Do we have any good choices here?

I've added some discussion of a direction we could pursue, using UAX#39's algorithm to form a fingerprint of identifiers to detect when they're unacceptably visually similar. Given that kind of approach, I think we should be free to use the full character set suggested by UAX#31 and only diagnose actual homoglyph collisions. Is that enough to satisfy your concern here?

[je4d]

Do we currently have, or plan to have, raw literals in Carbon, and if so do we need to consider an exemption to the requirement of valid UTF-8 for the content of such literals?
I'm imagining use cases such as a [local encoding]->utf8 translation unit test wanting to have a literal for its input.

[geoffromer]

FWIW I thought about raising the same question, but came to the conclusion that in such situations, you really should use something like unicode escape sequences rather than literal data. Otherwise you wind up in a situation where your source code has semantically-relevant properties that are completely invisible unless you open your source code in a hex editor, which seems like a really bad place to be.

[zygoloid]

We do plan to have raw literals, but I don't think we intend for them to be used to support arbitrary binary data in source files. I'm inclined to say that the C++ rules that retroactively undo some parts of the early phases of translation for raw literals are a mistake -- we should view raw literals as a way to include special characters such as \ and " in string literals more easily, and anything beyond that should be considered out-of-scope. (Newline characters are also a concern here; the string literals proposal suggests having single-line and multi-line string literals, orthogonal to raw and non-raw literals.)

For an encoding test of the kind you describe, I think the most reasonable source encoding (from a source readability point of view and to ensure that the source file is stable when processed with tools -- and that the literal contents don't get replaced with U+FFFDs) would be to use escape sequences or similar to express the input, and language rules that largely force that choice might even be a good thing.

[zygoloid]

@je4d I consider this thread to be resolved. Please can you close it if you agree?

[jfbastien]

There's a proposal for something related in C++: https://isocpp.org/files/papers/P2194R0.pdf
It would be good to align both.

Oops, I had these comments un-submitted since Friday, forgot to click the github button. Sorry :(

[jfbastien]

Why exclude comments? It seems like there's already checking for valid codepoints in comments (is there?), so requiring normalized comments isn't really more work: just change checking in strings.

Further, I think you can also require that strings be normalized too. If a developer wants a non-normalized string (valid usecase!) then they ought to explicitly escape the sequence in the string (i.e. spell out that they want 'letter followed by accent', not 'letter with accent'). I think this simplifies specification of valid source (it all needs to be normalized), and makes the code clearer because it's always explicit that non-normalized strings were meant (because you have to spell it out). You never allow accidental non-normalized. Further, editors probably normalize the entire file anyways, so purposefully using non-normalized strings likely break.

[zygoloid]

These are reasonable points, and we've not really explored this axis of choice. Counterpoints: requiring string literals and comments to be normalized will increase compilation times, and at least for comments, we don't seem to get much practical benefit.

[chandlerc]

I increasingly like this point.

I think that having string literals which are the same be guaranteed to have the same byte sequence in the source code is good.

And I think having string literals that form a different byte sequence be visually distinct is also good. See the discussion around C vs. KC and homoglyphs below.

[zygoloid]

The document was revised in the suggested direction; I consider this resolved.

[jfbastien]

Please add a reference: http:https://wg21.link/P1949

[zygoloid]

Done.

[jfbastien]

Does this align with http:https://wg21.link/P1949 ?

[zygoloid]

P1949 is proposing that C++ follows Unicode Annex 31, so to the extent that later Carbon proposals do as this proposal suggests, yes.

[jfbastien]

Right, I just want to have a reference to show the alignment to what C++ is doing, and to say that it's on purpose.

[zygoloid]

Note added, with a link to the C++ proposal.

[chandlerc]

Reference the discussion around homoglyphs below here? It seems relevant. See my comments there.

[zygoloid]

Cross-reference to the homoglyphs section added.

[chandlerc]

I feel like this should also reference some amount of the similar-appearing cases handled by the compatibility normalization forms. My understanding is that these are distinct from the "confusable" set of glyphs defined by Unicode (UAX#39 below).

Essentially, if we care about homoglyphs, i feel like we should also care about the compatibility normalization.

I can see two approaches here:

1. Solve through normalization
2. Solve through detection and rejection of collisions (as suggested here)

I feel like we should take a similar approach for both of these.

If we solve through normalization, we should use normalization form KC, but either canonicalize or directly reject even the use of confusable glyphs (they're always available via escape codes).

If we solve through detection, I think we should detect both KC != C cases, as well as confusables.

I'm worried about the cost of detecting KC != C cases sadly. As a consequence, I wonder if we should use normalization form KC specifically justified by our desire to not have potential confusion similar to that of homoglyphs. And I wonder if we should simply reject confusable characters completely and require their usage to route through escape codes. This approach would have the advantage of moving us further towards the byte sequence of string literals being directly reflected visually as well as differences in identifiers being reliably detected visually.

I'm not even suggesting we have to commit to a specific thing around the confusables here, but I think we need to pick a strategy as that seems likely to influence the normalization form desired.

[mconst]

I don't think compatibility characters are going to be a problem. ICU's confusability-detection routines already check for them, and also, nearly all those characters are things we can just ban because you shouldn't be using them in code anyway.

I'd love to make string literals visually unambiguous, so you can determine their byte sequence by looking at them. In particular, this is important for strings that are going to be interpreted by code, like shell commands or HTTP headers. Banning confusable characters outright is pretty heavy-handed, though -- it would make it nearly impossible to write text in Russian, for example, because so many Cyrillic letters are confusable with Latin letters.

Also, it'll be a nontrivial task just to enumerate all the characters that might be confusable when used in a string literal. As far as I know, most real-world confusability work so far has just dealt with URLs; string literals allow a broader set of characters, like spaces and more kinds of punctuation, so there are going to be new opportunities for confusion that haven't been explored. And even in the limited world of domain names, where a lot of work has gone into preventing spoofing, people discover new spoofing opportunities pretty regularly. This isn't a solved problem.

In general, I think normalization isn't what we should be focusing on here. Normalization is easy, well-understood, and largely irrelevant in practice because most real-world text is NFC anyway; whatever normalization option we pick will be fine. Confusability detection is difficult, poorly understood, and causes security problems. That's the part we need to think about.

[chandlerc]

FWIW, the proposal largely sets this aside for now, so there doesn't seem to be anything more to do here...

I do continue to think if we want to do anything for confusability in the language, we'll need to do it through some kind of "normalization", although clearly the ones defined by Unicode are not sufficient to address most confusability concerns.

I think just looking at language level confusability through the normalization lens helps make clear the scope we can realistically handle: we could probably normalize away accidentally pasted ligatures and similar things, but maybe not much else. Whether that is worth doing from a QoI perspective (its clearly insufficient for any kind of security) is something I think we don't currently know, but also I don't think needs to be decided right away, so I'm fine proceeding with the proposal as is.

[chandlerc]

I increasingly like this point.

I think that having string literals which are the same be guaranteed to have the same byte sequence in the source code is good.

And I think having string literals that form a different byte sequence be visually distinct is also good. See the discussion around C vs. KC and homoglyphs below.

[chandlerc]

(I just want to say my original comment has been addressed... didn't really want to resolve the thread because of @mconst 's comments)

[chandlerc]

Note that I think the complexity is still significantly reduced by requiring normalization. That means the compiler can take an approach such as a fallback path that renormalizes and re-starts lexing (or other potentially costly fallback strategy) rather than needing to normalize in the "expected" path.

But perhaps the greatest reason to require it is to get diff stability and byte-equivalence for things like string literals.