Stringapalooza

[StefanKarpinski]

0.5 Major tasks

Round 1

• Base: merge ASCIIString, UTF8String and ByteString into String, replace ASCIIString & UTF8String with String #16058
• Compat: add String, String: alias for new String or old UTF8String Compat.jl#192

Round 2

• Base: replace utf8, bytestring and string with String, replace bytestring with String #16453, deprecate utf8 for String #16469
• Base: replace s = ascii(s) with s = String(s); isascii(s) || error(...), ascii: only support checking String for pure ASCIIness #16396

Round 3

• Base: cleanup conversion mess to and from string types String constructor can segfault #16470, String constructor is bad #16713, restore non-copying String behavior, add unsafe_string*, ... #16731
• Base: provide a way to interact with Windows APIs, via Cwstring deprecate WString and wstring #16975, Base.transcode to replace utf8to16 and utf16to8 #16974
• make package with ASCII, Latin-1, UTF-8, UTF-16 and UTF-32 string types (LegacyStrings).
• Base: remove UTF16String and UTF32String remove UTF-16 and UTF-32 stuff #16590

Cleanup tasks

• figure out LineEdit test change, replace ASCIIString & UTF8String with String #16058 (comment), fix: fix mystery REPL test change introduced by 5de52cf9c9343cfcf50be4c #16198
• delete redundant String inner constructor, replace ASCIIString & UTF8String with String #16058 (comment)
• undelete "redundant" String inner constructor, replace ASCIIString & UTF8String with String #16058 (comment)
• replace jl_is_utf8_string and jl_is_byte_string with jl_is_string, replace ASCIIString & UTF8String with String #16058 (comment)
• figure out code coverage drop, replace ASCIIString & UTF8String with String #16058 (comment)
• use ascii in uv_getaddrinfo to error on non-ASCII domain names, replace ASCIIString & UTF8String with String #16058 (comment)
• use ascii in SuiteSparse to error on non-ASCII inputs,
  replace ASCIIString & UTF8String with String #16058 (comment)
• remove special case printing for String type, 5de52cf#commitcomment-17373792, Fix printing of TypeVar with lower bound. #16221
• make sure tests pass with inlining off, replace ASCIIString & UTF8String with String #16058 (comment)
• add deprecation for readdlm's ignore_invalid_chars option, replace ASCIIString & UTF8String with String #16058 (comment), e95f5f2
• remove documentation for readdlm's ignore_invalid_chars option 969d61b
• NEWS.md entries
• fix signature of connect in docs,
• update doc/manual/interacting-with-julia.rst (with doctests), replace ASCIIString & UTF8String with String #16058 (comment)
• add iconv tests JuliaStrings/LegacyStrings.jl#4

---

0.6 Major tasks

Round 4

• Base: change Char representation (allow lossless string processing of any data)
• Base: remove RepString (moved to LegacyStrings)
• Base: remove RevString (move to package?)
• Base: merge SubString and String (add offset field to String)

Cleanup tasks

• make prevind("ll", 5) and such errors, replace ASCIIString & UTF8String with String #16058 (comment)
  replace ASCIIString & UTF8String with String #16058 (comment)
• improve isspace implementation, replace ASCIIString & UTF8String with String #16058 (comment)
• figure out why removing seemingly redundant convert method breaks bootstrap, replace ASCIIString & UTF8String with String #16058 (comment)
• simplify takebuf API, replace ASCIIString & UTF8String with String #16058 (comment), Simplify takebuf() API #19088
• move docstrings inline

[JeffBezanson]

Great list.

What about windows APIs that use utf-16?

[StefanKarpinski]

What about windows APIs that use utf-16?

Already taken care of: #15033.

Two more potential rounds:

• removing RepString and RevString
• merging SubString and String

Might not get to those until the next release though.

[quinnj]

What about:

• String iteration: making this faster, based on your experiments here in the past
• Decide on String indexing: Restrict indexing into strings to a special ByteIndex or StringIndex type #9297

I definitely think there's a lot to play with around merging Substring and String, but it certainly feels like 0.6 material.

[tkelman]

It was just added, but now that readstring(io) is just String(read(io)) it maybe doesn't even need a separate name forever

[lobingera]

just for my curiosity: Why should this be part of 0.5? The title of the release had something to do with Arrays ... not Strings?

[nalimilan]

+1 for including as much of this plan as possible into 0.5. Breakage better happen soon.

As regards moving ASCIIString, etc. to a package (round 3, bullet 2), see discussion on an implementation for any encoding in StringEncodings.jl here.

As regards changing Char's underlying representation(round 4, bullet 1), another step would also be to introduce AbstractChar and use it in method signatures to allow e.g. the ASCIIString replacement to implement ASCIIChar <: AbstractChar as a UInt8. This would allow people working with ASCII data to actually enjoy higher performance than before.

EDIT: Finally, I'd like to see a discussion regarding the opportunity of ensuring that String only holds valid UTF-8 data or not, and how to handle file paths (special string type or not). But I don't want to derail this already rich thread, so maybe better open a separate issue for that?

[StefanKarpinski]

just for my curiosity: Why should this be part of 0.5? The title of the release had something to do with Arrays ... not Strings?

Because I've been working on this for months and it's ready to go. It won't hold up the release anyway.

[lobingera]

I somehow agree, it will not hold the release of the language. But more syntax changes in 0.5 give some impact in porting packages to 0.5; maybe i'm just wrong that this causes effort ...

[nalimilan]

I somehow agree, it will not hold the release of the language. But more syntax changes in 0.5 give some impact in porting packages to 0.5; maybe i'm just wrong that this causes effort ...

And more syntax changes in 0.6 will cause effort for even more packages. And given the growth rate of the package ecosystem...

[stevengj]

#15033 didn't fully provide a path for external packages to call Windows APIs, without calling internal functions.

[StefanKarpinski]

True – added to the roadmap.

[StefanKarpinski]

Actually, question here: do people think we should keep ascii and have it convert strings to standard String type and error if the content is not plain ASCII? It's kind of a useful function to have.

[StefanKarpinski]

Another question about behavior. String and string don't actually behave in the same manner always:

julia> String(UInt8[97,98,99])
"abc"

julia> string(UInt8[97,98,99])
"UInt8[97,98,99]"

Any thoughts on resolving this? Currently utf8 and ascii behave like String not like string.

[stevengj]

Maybe String! if it takes ownership of the array, i.e. is "in-place"?

There is also the bytestring function, whose name seems like a holdover from ByteString, but whose function is essential.

[StefanKarpinski]

I'm pretty sure that usage of ! does not have the @JeffBezanson seal of approval.

[stevengj]

String¡, then.

[JeffBezanson]

This case doesn't bother me that much --- unlike utf8 and ascii, string is not an encoding.

[stevengj]

The point is, we need some function to replace bytestring(::Vector{UInt8}) and bytestring(::Ptr{UInt8}, [len]) (makes a copy of bytes), but also UTF8String(::Vector{UInt8}) and pointer_to_string (doesn't make a copy).

It makes sense to me to name them all the same thing, with ! for the "in-place" versions. But I don't know what that name should be. Maybe bytestring and bytestring!, where byte refers to the encoding? Or utf8 and utf8! to be more explicit about the encoding, with ascii doing an additional assert?

[JeffBezanson]

Can we just keep bytestring?

[stevengj]

@JeffBezanson, bytestring is fine if a bit vague, but it always makes a copy. Would you be okay with bytestring! for the non-copying version?

[JeffBezanson]

I'm not prepared to make this the first ever case where ! means something other than mutation.

[stevengj]

Then what do we call the non-copying version(s) of bytestring?

[JeffBezanson]

I've probably missed some of the discussion here, but can String have the same constructors that UTF8String had? Plus one with a Ptr argument?

[stevengj]

@JeffBezanson, the problem is that then it can't replace string.

[JeffBezanson]

That's fine with me. Somehow we need one function that wraps a UInt8 vector as a string, and another that gives you the output of print as a string. We could rename string to something like sprint, except that's taken.

[StefanKarpinski]

TODO: Cstring and Cwstring should ensure that string data is NUL terminated (as well as NUL free).

[StefanKarpinski]

Ref #16499. Also need a way to express conversion of String to non-NUL-terminated UTF-16 data with known length. Perhaps convert(Vector{UInt16}, s)?

[stevengj]

@StefanKarpinski, in previous incarnations, any UInt8 array allocated by Julia was automatically NUL-terminated internally; is this no longer the case? And UTF16String and UTF32String were NUL-terminated, so convert(Cwstring, string) was also.

[JeffBezanson]

Which items are slated for 0.5? Through round 3, IIRC?

[StefanKarpinski]

Yes, that's correct. Tomorrow/Wednesday I need to create a LegacyStrings package, put all the Unicode stuff in it and then merge my PR that removes all of that stuff with deprecations that point at it.

[JeffBezanson]

PR is up to remove RepString; it has already been added to LegacyStrings. RevString is used by some Base functions so we might want to leave it for now. Anything else here planned for 0.6?

[StefanKarpinski]

While you're doing string stuff, it's probably not too hard to just actually do utf-8 reversal on strings instead of using the RevString type – that would advance the highlander agenda just a bit further. (If you feel like it and have some spare type while waiting for type revamp test to run or something.)

[StefanKarpinski]

Other than removing RevString everything that's likely to be done is already done.