Quotes and string interpolation

[friggeri]

The shortest complete expression after the $ is taken as the expression whose value is to be interpolated into the string. Thus, you can interpolate any expression into a string using parentheses:

This doesn't work :

julia> "$("string")"
syntax error: macro string_str not defined

I suspect a lexing/parsing error, it gets parsed as ["$"] [string")" ].

[JeffBezanson]

My first thought was that it should be written "$(\"string\")", but that doesn't work either since it seems unescaping isn't done on the part that's parsed. That was probably intentional, since it means in "$(a\n)\n" the first backslash is a backslash operator and the second is a newline escape sequence. It might be simpler to uniformly apply unescaping everywhere first though.

Alternatively, if we make "$("string")" work, then it won't be possible to find the end of strings without a full julia parser, which does not bode well for editor modes.
But @StefanKarpinski will have to decide.

Personally I say all this stuff is just an inscrutable way to do something totally trivial, much like shell syntax. You can write hundreds of lines of code to implement it and it will still never work.

[friggeri]

As a reference, it does work in coffeescript (and is correctly highlighted at least in vim and textmate) :

"#{"string"}" # "string"

[JeffBezanson]

Probably possible to handle 99% of cases with simple editor code.
But we can have curly braces and backslashes as part of expressions, which makes it harder.

[friggeri]

I'd rather have it wrongly highlighted in my editor in 1% of cases than having to introduce another variable juste to be able to put a string inside an interpolation. I've had this problem today :

"[$(join(arr, " - "))]" #doesn't work

# this works but is ugly :
tmp = join(arr, " - ")
"[$tmp]"

[JeffBezanson]

I agree. Stefan owns this functionality; we'll just have to wait until he is available again.

But, I would have used "[" * join(arr," - ") * "]".

[StefanKarpinski]

Sigh. I knew I'd have to face the music on this one eventually. So let me explain why this problem exists first...

The way string interpolation works is actually entirely defined in Julia. What happens is that the parser (in FemtoLisp) scans the code and finds a string literal, delimited by double quotes. If it finds no unescaped $ in the string, it just creates a string literal itself — ASCIIString or UTF8String depending on the content of the string. On the other hand, if the string has an unescaped $, it punts and hands the interpretation of the string literal to the str julia macro, which generates an expression that constructs the desired strings by concatenating string literals and interpolated values. This is a nice elegant scheme that lets the parser not worry about stuff like interpolation.

The problem is that this scheme separates the parsing of the embedded subexpression from the determination of what actually constitutes the string literal in question. So how can the parser know where the string literal ends in something like "$("foo")". It can't really. We do this sort of thing a lot. It's how backticks for command literal work too. It doesn't really come up there, however, because splicing a command literal into a command literal is a bizarre thing to do that I can't concoct a good reason for. Splicing string literals into strings, however, is pretty reasonable.

Here's the only solution I can think of that allows us to keep the current elegant separation of concerns, and it actually rather resembles how our parser interacts with the repl. Basically, instead of just failing or succeeding when the parser passes a string literal to the str macro, there need to be three possible results: fail, succeed, or incomplete. The incomplete version indicates that the string literal with interpolation that was given isn't complete — the parser would then have to try again with more string. So for example, this would be the process for parsing "$("foo")":

1. Parser tries @str "\$(", the macro returns incomplete;
2. Parser tries @str "\$(\"foo", the macro returns incomplete;
3. Parser tries @str "\$(\"foo\")", and the macro returns success.

This last attempt produces the desired syntax tree for constructing a string via interpolation. @JeffBezanson, do you think this can be made to work?

[JeffBezanson]

No, that's insane. The only simple solution is to have the lexer invoke the parser as soon as it sees a "$" in a string, then resume normal string reading after the parser returns. I don't like this, because then you need the full parser to know what lexically constitutes a string.

More of a compromise is to make the syntax "$(\"foo\")", i.e. you still need to use string escapes inside $. This only affects quotes and backslashes, and interpolating an expression involving backslash is pretty strange. Basically just call unescape on the whole string before dealing with $.

[StefanKarpinski]

Having the lexer invoke the parser will work for the specific case of $-interpolation in standard strings but not for any kind of custom string that anyone wants to write. My scheme will work for anything. So the upshot of doing it your way would be that non-standard string literals cease to be first-class, in the sense that they couldn't, e.g., implement interpolation the way standard Julia string literals do. I don't really see why my scheme is so insane.

The compromise of "$(\"foo\")" would work for any kind of string literal but it's just ugly.

[JeffBezanson]

Just so we're clear, we're effectively talking about read-macros now: custom julia code that runs during lexing. How are we going to support syntax highlighting in that case?

IMO, the language already provides perfectly decent syntax. String interpolation throws it away in exchange for metacharacter hell.

[StefanKarpinski]

That's kind of true — but very limited read-macros. The escaping of quotes inside quotes just seems too awful to me. I'd rather just not allow double-quoted expressions to be interpolated into strings.

[StefanKarpinski]

I have an idea for this. If we implement #70, then we can trivially allow quotes within quotes: """$("foo")""". So instead of quoting the inner quotes, you have to just use triple-quotes for the outer quotes instead. @afriggeri: would that be satisfactory? I've never found it very palatable that you potentially need a full language parser to know when quotes with interpolation end in many languages.

[JeffBezanson]

Sounds good to me. Close?

[nolta]

Status changed from "won't fix" to "fixed".