A set of naive tools to convert between broken UTF-16 and WTF-8. See https://en.wikipedia.org/wiki/UTF-8#WTF-8

The only purpose of these tools is to convert to and from broken UTF-16 (that is, with unpaired surrogates), which Windows seem to happily generate.

Basically, all it does is happily read or write unpaired surrogate halves.

wtf162wtf8 reads UTF-16 code units, and tries to read code points. If that succeeds, write the read code point as UTF-8. If it doesn't succeed, i.e. if it is a high or low surrogate without its other half, write the surrogate half as UTF-8 (which makes it WTF-8).

The result is WTF-8, and even UTF-8 if the input is valid UTF-16.

wtf82utf16 does the revers conversion: given WTF-8 input, it reconstructs the possibly broken UTF-16 data. All it does is actually write every code points below 0x10000 as plain UTF-16 units, even surrogate halves.

As a proof of concept, there is also support for broken UTF-32. Just like WTF-8 and broken UTF-16, is allows reserved code points to appear and encodes and decodes them happily. Only WTF-8/UTF-32 pairs are provided, but they can be streamed together to convert directly between UTF-16 and UTF-32, using e.g. wtf162wtf8 < input | wtf82utf32 > output.

These tools are naive, and don't actually do anything about endianess. The result is that if they are run on a Big Endian machine, they read and write UTF-16BE, and if they are run on a Little Endian machine (fairly more common), they read and write UTF-16LE.

As those tools are typically useful with UTF-16LE, and most machines are Little Endian, it should generally work fine. Hopefully.

To convert from (broken) UTF-16 to WTF-8, use wtf162wtf8 < input > output. Similarly, to convert from WTF-8 to (broken) UTF-16, use wtf82utf16 < input > output.

You can control the verbosity through the VERBOSE environment variable: set it to a positive integer to get verbose/debugging output on stderr.