Strings::webalize() returns an empty string on Alpine Linux

[mochja]

I think this is due to musl implementation of iconv.

bash-4.3# vendor/bin/tester tests -p php -c /etc/php/php.ini
 _____ ___  ___ _____ ___  ___
|_   _/ __)( __/_   _/ __)| _ )
  |_| \___ /___) |_| \___ |_|_\  v1.7.1

PHP 7.0.6 | php -c '/etc/php/php.ini' | 8 threads

ss...............................sssssss.Fs...............F......................F......F......................F.F......

-- FAILED: Nette\Utils\Json::encode() | tests/Utils/Json.encode().phpt
   Failed: Nette\Utils\JsonException was expected, but none was thrown

   in tests/Utils/Json.encode().phpt(39) Tester\Assert::exception()

-- FAILED: Nette\Object undeclared method. | tests/Utils/Object.undeclaredMethod.phpt
   Failed: Nette\MemberAccessException was expected but got Nette\NotSupportedException (PHP extension GD is not loaded.)

   in tests/Utils/Object.undeclaredMethod.phpt(52) Tester\Assert::exception()

-- FAILED: Nette\SmartObject undeclared method. | tests/Utils/SmartObject.undeclaredMethod.phpt
   Failed: Nette\MemberAccessException was expected but got Nette\NotSupportedException (PHP extension GD is not loaded.)

   in tests/Utils/SmartObject.undeclaredMethod.phpt(54) Tester\Assert::exception()

-- FAILED: Nette\Utils\Strings::chr() | tests/Utils/Strings.chr().phpt
   Exited with error code 255 (expected 0)
   E_NOTICE: iconv(): Wrong charset, conversion from `UTF-32BE' to `UTF-8//IGNORE' is not allowed

   in src/Utils/Strings.php(57)
   in src/Utils/Strings.php(57) iconv()
   in tests/Utils/Strings.chr().phpt(14) Nette\Utils\Strings::chr()

-- FAILED: Nette\Utils\Strings::toAscii() | tests/Utils/Strings.toAscii().phpt
   Exited with error code 255 (expected 0)
   E_NOTICE: iconv(): Wrong charset, conversion from `UTF-8' to `ASCII//TRANSLIT//IGNORE' is not allowed

   in src/Utils/Strings.php(185)
   in src/Utils/Strings.php(185) iconv()
   in tests/Utils/Strings.toAscii().phpt(14) Nette\Utils\Strings::toAscii()

-- FAILED: Nette\Utils\Strings::webalize() | tests/Utils/Strings.webalize().phpt
   Exited with error code 255 (expected 0)
   E_NOTICE: iconv(): Wrong charset, conversion from `UTF-8' to `ASCII//TRANSLIT//IGNORE' is not allowed

   in src/Utils/Strings.php(185)
   in src/Utils/Strings.php(185) iconv()
   in src/Utils/Strings.php(201) Nette\Utils\Strings::toAscii()
   in tests/Utils/Strings.webalize().phpt(14) Nette\Utils\Strings::webalize()


FAILURES! (120 tests, 6 failures, 10 skipped, 1.1 seconds)

[dg]

Can you inspect it or try to find solution? Related #107

[mochja]

There is no transliteration support at all.

[Majkl578]

I guess your best luck would be by installing php-intl, will that help?
But I think there should be an exception thrown in this case, when no suitable conversion backend is found.

[mochja]

List of all installed modules, including intl

[PHP Modules]
bcmath
Core
ctype
curl
date
dom
fileinfo
filter
hash
iconv
intl
json
libsodium
libxml
mbstring
memcached
mysqlnd
openssl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
readline
Reflection
session
SimpleXML
SPL
sqlite3
standard
tokenizer
xml
xmlreader
xmlwriter
zip
zlib

@dg possible solution https://github.com/symfony/polyfill-iconv

[ludekvesely]

I think there is not problem in Nette but in Alpine Linux. symfony/polyfill-iconv didn't work correctly, I had to modify it: ludekvesely/polyfill-iconv@246a751. It's a workaround but it works for me. And I agree that there should be notice/exception thrown in this case.

For me these steps worked:

1. remove (or do not install) iconv

apk del php7-iconv

2. update composer.json

...
"repositories": [
    {
        "type": "vcs",
        "url": "https://github.com/ludekvesely/polyfill-iconv.git"
    }
],
"require": {
    ...
    "symfony/polyfill-iconv": "dev-fix-ignore-order"
}
...

3. install polyfill

composer update

[Majkl578]

Have you reported it to upstream?

[dg]

It triggers E_NOTICE so it is IMHO ok, or not?

[ludekvesely]

I think E_NOTICE is ok.

[enumag]

@dg In my opinion it is not ok. On localhost it works fine for me because I've glibc iconv. However in docker it doesn't. Obviously docker has production settings so nobody knew about the E_NOTICE. In my opinion we should fallback to iconv('UTF-8', 'ASCII', $s) if iconv('UTF-8', 'ASCII//TRANSLIT//IGNORE', $s) returns an empty string and the string before calling iconv is non-empty.

Alternatively we could use iconv('UTF-8', 'ASCII', $s) if ICONV_IMPL is unknown.

Also Strings::chr() suffers from this bug as well because 'UTF-8//IGNORE' doesn't work either.

Leaving it as is means that I can't use toAscii or webalize anywhere in my application and have to check all 3rd party libraries that they don't use them either. And that is really not ok.

[Majkl578]

@enumag It's not a Docker issue, it's just an issue with the Alpine image. Use different base image that is not intentionally light and missing essential features.

[enumag]

@Majkl578 Well Nette already has some workaround for glibc, there is no reason for it to not have another workaround for musl (which is used in alpine) if such workaround is possible. If it is not possible I'd expect an exception that musl implementation of iconv is not supported by Nette instead of just E_NOTICE and this should be checked by Nette requirements checker as well. Returning completely wrong values with just a notice is a bug in my opinion.

[Majkl578]

I've never heard of a thing called musl until now and as far as I know there's barely any Linux distro that uses it. I'd be just fine with calling it unsupported (at least for now).

[enumag]

Alpine is used quite often to have docker images small. Even if musl is used by Alpine alone it is enough to introduce some support for it or at least throw a proper exception.

[plispe]

@enumag I found a solutions for this problem with iconv //TRANSLIT and //IGNORE flags. You just build php-iconv extension with GNU libiconv. You can see it in this Dockerfile. I tried it today with php 7.0.19 and it works like a charm.

[enumag]

@plispe Thanks but I'm well aware of that solution. My point is this workaround should not be needed or there should be an exception to let me know there is something wrong. Current solution with silent error is insufficient for a library like Nette.

[dg]

@enumag so send PR.

[fprochazka]

@dg fyi, having you react and confirm sending a PR won't be a waste of time (and won't be rejected right away because of the idea, not wrong implementation) is really helpful. </no-sarcasm>

[plispe]

@enumag sure, I understand your argument and I agree, that Nette should throw an exception in this case. With my solution(workaround), you can use //TRANSLITERATE and //IGNORE Iconv flags on alpine linux, but it should not be needed in this case.

[enumag]

I tried but unfortunately I don't know what to use instead of //TRANSLIT and //IGNORE to achieve the same result. Any ideas?

Also does anyone know what other implementations of iconv are out there? ICONV_IMPL on Alpine is 'unknown'.