Composition bug

[KrokodileGlue]

It appears that some code point pairs are being incorrectly composed; a starter followed by a non starter seems to always produce a composition, even if the logical resulting character doesn't exist. For instance, the sequence U+72 U+307 U+323 represents a lowercase latin r with a dot below and a dot above. There does exist LATIN SMALL LETTER R WITH DOT BELOW, so the composition of the first pair results in a valid character. The composition of the resulting code point (U+1E5B) together with U+323, however, results in U+1E64 (LATIN CAPITAL LETTER S WITH ACUTE AND DOT ABOVE) because there is no LATIN SMALL LETTER R WITH DOT BELOW AND DOT ABOVE.

Many other sequences produce errors. All of the following compositions are incorrectly produced by utf8proc, but of course there are infinitely many more:

• U+61 + U+307 = U+227
• U+227 + U+323 = U+2E
• U+227 + U+307 = U+1FC
• U+1E5B + U+323 = U+1E64

The actual character that's produced in error, of course, is the result of a reasonable index into the data table, so I doubt any check on the value of composition could reliably fix this.

Here's a minimal program demonstrating the bug:

#include <stdio.h>
#include <utf8proc.h>
int main(void)
{
	char *src = "\x72\xCC\x87\xCC\xA3";
	char *norm = (char *)utf8proc_NFC((utf8proc_uint8_t*)src);
	puts(src);
	puts(norm);
	return 0;
}

[stevengj]

Thanks, I can reproduce the bug.

[stevengj]

It looks like an earlier version of utf8proc didn't have this bug, so it would be extremely helpful if you could do a bisect.

(Happened somewhere between utf8proc 1.3 and 2.0.2, it seems?)

[KrokodileGlue]

It looks like this was introduced by eeebf70. I think the relevant change is "merge 1st and 2nd comb index (saves 50kb)". This particular change may have to be reverted and the tables regenerated.

[stevengj]

@benibela, can you look into this?

[stevengj]

Closed by #129.