[Bug] Character encoding error in the name of the German IPTV channel 'MDR Thüringen'

[mendel5]

Today I noticed a character encoding error in the name of the German IPTV channel 'MDR Thüringen'. The German letter Ü in the second word is displayed incorrectly. The Ü is also called U-umlaut, see https://en.wikipedia.org/wiki/%C3%9C.

'MDR Thüringen' is the public service television broadcaster for the German state Thuringia, see https://en.wikipedia.org/wiki/Mitteldeutscher_Rundfunk .

The channel can be found here:

• Start Hypnotix
• Click on TV Channels (3607)
• Click on GERMANY (34)

Here is a screenshot of the character encoding error:

[mendel5]

Here are some more errors:

Should be allgäu.tv Kempten, see https://de.wikipedia.org/wiki/Allg%C3%A4u.tv

Should be:

• Filstalwelle Göppingen, see https://de.wikipedia.org/wiki/Filstalwelle
• Franken Fernsehen Nürnberg, see https://de.wikipedia.org/wiki/Franken_Fernsehen

[mendel5]

I noticed there is another issue about the same topic here: #101 .
Maybe you @Free-IPTV can have a look at this? Thanks!

[Free-IPTV]

I noticed there is another issue about the same topic here: #101 .
Maybe you @Free-IPTV can have a look at this? Thanks!

Hi @mendel5,

The problem has long been resolved, actually. Sometimes there were problems with umlauts (ä, ü and so on) but then all m3u's were saved in UTF-8 format, then that was done. Could it be that you still use my old m3u's? All m3u's in my repo are now saved in UTF-8 format.

[mendel5]

Could it be that you still use my old m3u's?

I don't think so. To be sure I removed and reinstalled hypnotix with these commands:

$ sudo apt purge hypnotix
$ sudo apt-get install hypnotix

My hypnotix version is 1.5.

In my Providers settings there is one provider (Free-IPTV) with the URL pointing to https://raw.githubusercontent.com/Free-IPTV/Countries/master/ZZ_PLAYLIST_ALL_TV.m3u. I removed this provider entry and added it again so that the m3u file is getting updated.

Unfortunately that did not solve the issue. The names of some German IPTV broadcasters are still displayed with encoding errors.

[Free-IPTV]

Unfortunately that did not solve the issue. The names of some German IPTV broadcasters are still displayed with encoding errors.

Hmm, okay, problem is not solved. I don't know which character sets Hypnotix supports. I use Kodi myself, so the ZZ_PLAYLIST_ALL_TV.m3u is created with the Add-ON IPTV-Merge. As I wrote, we used to have the problem with special characters, but since I have been saving the m3u's in UTF-8 format, there have been no more problems.

Of course this has to be solved somehow, how does Hypnotix deal with character codes, is UTF-8 supported?

[mendel5]

Hmm, okay, problem is not solved. I don't know which character sets Hypnotix supports.
how does Hypnotix deal with character codes, is UTF-8 supported?

I'm quite sure that Hypnotix supports UTF-8. When I open the file ZZ_PLAYLIST_ALL_TV.m3u in Linux Mint's xed text editor, I get this error:

However when I open the file DE01_GERMANY.m3u where MDR Thüringen is included there is no encoding error:

So this issue seems to be about the specific file ZZ_PLAYLIST_ALL_TV.m3u and not about Hypnotix.

[mendel5]

I just did another test to check whether the issue arises from the file or from Hypnotix.

In Hypnotix' Providers overview I deleted the entry with the URL pointing to https://raw.githubusercontent.com/Free-IPTV/Countries/master/ZZ_PLAYLIST_ALL_TV.m3u and created a new provider entry with the URL pointing to https://raw.githubusercontent.com/Free-IPTV/Countries/master/DE01_GERMANY.m3u.

When using the second file DE01_GERMANY.m3u the IPTV broadcaster MDR Thüringen is displayed correctly (without character encoding error).

So by now I'm sure this issue has something to do with the file ZZ_PLAYLIST_ALL_TV.m3u.

[Free-IPTV]

Okay, when you open the Playlist here, there all characters okay:

https://raw.githubusercontent.com/Free-IPTV/Countries/master/ZZ_PLAYLIST_ALL_TV.m3u

In other apps, the problem with the playlist no longer exists since all files (including the playlist) are stored in UTF-8.

The only thing I can do now, I swap all letters (special characters), as an example Ü for ue and so on. But that takes time and is not even done in a few minutes. I could start doing that tomorrow morning. I just don't understand it, since the playlist is also in UTF-8 format and nobody has reported a problem with character codes since then.

[mendel5]

The only thing I can do now, I swap all letters (special characters), as an example Ü for ue and so on.

No, please don't do that. I will try to fix this and get back to you.

[Free-IPTV]

The only thing I can do now, I swap all letters (special characters), as an example Ü for ue and so on.

No, please don't do that. I will try to fix this and get back to you.

Tomorrow Morning i create with the newest IPTV-Merge Addon and Kodi 19 a new Playlist and we test it, okay? I has a second TV-Box with 905x3 to test some things in the new Kodi 19. ;-)

[mendel5]

Tomorrow Morning i create with the newest IPTV-Merge Addon and Kodi 19 a new Playlist and we test it, okay?

Sure, we can do that.

What is the origin of the individual files (not the ZZ_PLAYLIST_ALL_TV.m3u)? In https://github.com/Free-IPTV/Countries/blob/master/README.md you write:

To automatically create the individual lists to one list, i use the ADD-On IPTV-Merge from Matt Huisman.

But where do the individual lists come from in the first place?

[Free-IPTV]

Tomorrow Morning i create with the newest IPTV-Merge Addon and Kodi 19 a new Playlist and we test it, okay?

Sure, we can do that.

What is the origin of the individual files (not the ZZ_PLAYLIST_ALL_TV.m3u)? In https://github.com/Free-IPTV/Countries/blob/master/README.md you write:

To automatically create the individual lists to one list, i use the ADD-On IPTV-Merge from Matt Huisman.

But where do the individual lists come from in the first place?

I have all the individual lists on my PC. If there are updates from me, I upload the new file to Github. In the case of updates from others, I also save the modified m3u on my PC. Check out the playlist here, it is now created with the latest IPTV merge under Kodi 18. Rename it to .m3u ;-)

ZZ_PLAYLIST_ALL_TV.txt

[mendel5]

I think I found the character encoding error. It's located in the file DE03_GERMANY_Local.m3u.

You can reproduce the process on your PC:

• Use GNU/ Linux, e.g. Linux Mint
• Install the package moreutils by executing the command $ sudo apt-get install moreutils in a terminal
• Open https://github.com/Free-IPTV/Countries and download the repository by clicking Code and Download ZIP
• Unpack the ZIP file on your local machine
• Open a terminal and navigate to the unpacked folder Countries-master by using cd
• Run the command $ isutf8 --verbose *

[Free-IPTV]

I think I found the character encoding error. It's located in the file DE03_GERMANY_Local.m3u.

Okay, i upload this file again to github tomorrow morning, when i makes the next updates. I use Windows, edit the files with Notepad++ which supports UTF-8. I has no Linux... sorry. ;-) Only Libreelec with Kodi on my Raspi4 ;-)

I'll get back to you tomorrow, okay?

Have a nice evening. :-)

[mendel5]

Yes, thank you.

I added a comment here: Free-IPTV/Countries@6a6a05a#r47965647.

Direct link to the line: https://github.com/Free-IPTV/Countries/blob/49ab8324f2a941e685279bb4c739a3705bc71ac8/DE03_GERMANY_Local.m3u#L6 (see end of line)

Here is the output of $ isutf8 --verbose *:

$ isutf8 --verbose *
DE03_GERMANY_Local.m3u: line 6, char 189, byte 578: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
6E 67 22 2C 61 6C 6C 67 E4 75 2E 74 76 20 4B 65  | ng",allg.u.tv Ke
                        ^^^^^                    |         ^^

ZZ_PLAYLIST_ALL_TV.m3u: line 400, char 189, byte 86973: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
6E 67 22 2C 61 6C 6C 67 E4 75 2E 74 76 20 4B 65  | ng",allg.u.tv Ke
                        ^^^^^                    |         ^^

You only need to fix the character encoding in the first file. Then you can merge all individual lists to the big one and the fix should be included as well.

[Free-IPTV]

Hi @mendel5,

I hope that all characters are now displayed properly. Thanks for your great search for help.

By the way, Iran joined last week, Pakistan joined today. We now have over 80 m3u's with over 3600 streams ... brilliant. :-)

[mendel5]

I hope that all characters are now displayed properly.

Yes, that seems to be the case.

Thanks for your great search for help.

Thanks for your quick response!

I will close this issue now.