Hugo cannot identity correct translation string when multiple variants of a language code is used

[shaform]

What version of Hugo are you using (hugo version)?

$ hugo version
Hugo Static Site Generator v0.78.2 linux/amd64 BuildDate: 2020-11-13T14:14:29Z

Does this issue reproduce with the latest release?

Yes.

Steps to Reproduce

The problem can be reproduced by following the steps below:

1. Copy the current multilingual example to local directory: https://github.com/gohugoio/hugo/tree/b8758de19ec75b4565075314f9578270a092bc6f/examples/multilingual

2. cd into the local directory and create zh files by executing the commands:

cp content/home.en.md content/home.zh.md
cp i18n/en.toml i18n/zh.toml
cp i18n/en.toml i18n/zh-cn.toml

3. Modify i18n/zh.toml, set other= Correct title

4. Modify i18n/zh-cn.toml, set other= Wrong title

5. Add the following content into config.toml

[languages.zh]
weight = 1
title = "Test ZH language"
[[languages.zh.menu.main]]
url = "/zh/home"
name = "ZH Home"
weight = 0

6. Execute hugo server.
   When hugo server is executed, a warning message is displayed, indicating something is wrong:

WARN 2020/11/24 10:10:59 Failed to get translated string for language "zh" and ID "head_title": %!s(<nil>)

7. Open browser and go to url HUGO_URL/zh/home. We should expect that the page title is Correct title. However, Wrong title is displayed as the page title.

Comments

This is probably related to #7838.

[past-due]

I believe I am also experiencing this issue, and it still exists in Hugo 0.86.0.

[past-due]

@bep: Did you happen to have any pointers / ideas on where the problem may lie? (Or is this more likely an issue with a dependency? I'd be happy to test out fixes, but have not had a chance to dig into the codebase yet.)

[smailzhu]

I think you can fix the problem by remove either zh.toml or zh-cn.toml

What's the difference between this language file? I thought there same?

[shaform]

@smailzhu That doesn't "fix" the problem. These files could certainly be different. Using another example to illustrate the point: maybe the site owner wants to have en.toml and en-au.toml, in which the first file contains the default English translations while the second file contains translations for a specific English-language dialect in Australia. It would defeat the purpose of having multilingual support if we just ask the user to delete the translations for different dialects.

[smailzhu]

About this question I think it may have some problem with i18n in hugo
issue #3564 also mention similar issue with nil pointer, does this problem caused by go-i18n?

[smailzhu]

@shaform I cannot reproduce the warning with en.toml and en-au.toml
it seems i18n somehow has problem with zh.toml and zh-cn.toml . It works without warning with zh-hant.toml + zh.toml, zh-hk.toml + zh.toml, zh-tw.toml + zh.toml, zh-hant.toml + zh-hk.toml + zh-tw.toml + zh.toml, or zh-hant.toml + zh-hk.toml + zh-tw.toml + zh-hans.toml

[shaform]

it seems i18n somehow has problem with zh.toml and zh-cn.toml .

That's interesting. I wonder what's so special about zh-cn? It seems nicksnyder/go-i18n#72 (which causes #3564) has already been fixed though.

[past-due]

I have seen this issue with pt + pt-BR as well.

[jmooring]

This was resolved in v0.112.0.

Minimal example:

git clone --single-branch -b hugo-github-issue-7982 https://github.com/jmooring/hugo-testing hugo-github-issue-7982
cd hugo-github-issue-7982
hugo

[McShelby]

There's something strange going on here.

I've taken your example, added a similar i18n/en.toml and changed the config.toml to this

baseURL = 'https://example.org/'
languageCode = 'en-US'
title = 'Hugo GitHub Issue #7982'

defaultContentLanguage = 'en'
defaultContentLanguageInSubdir = true

[languages.en]
contentDir = 'content/en'
languageCode = 'en'
languageName = 'English'
weight = 1

[languages.zh]
contentDir = 'content/zh'
languageCode = 'zh-CN'
languageName = 'Mandarin'
weight = 2

...and the error reappeared

WARN  Failed to get translated string for language "zh" and ID "foo": %!s(<nil>)

If I change defaultContentLanguage back to zh, the error is gone.

[jmooring]

I've update the test conditions per your remarks and can reproduce the problem.

git clone --single-branch -b hugo-github-issue-7982 https://github.com/jmooring/hugo-testing hugo-github-issue-7982
cd hugo-github-issue-7982
hugo

Expected result in console:

WARN  "foo in en"
WARN  "foo in zh"

Actual result in console:

WARN  "foo in en"
WARN  Failed to get translated string for language "zh" and ID "foo": %!s(<nil>)
WARN  "foo in zh-CN"

I suspect this is an upstream problem, but it will take some effort to dig into.

[jmooring]

hugo/langs/i18n/i18n.go

Lines 118 to 120 in 9dd6870

	if _, ok := err.(*i18n.MessageNotFoundErr); !ok {
	t.logger.Warnf("Failed to get translated string for language %q and ID %q: %s", currentLangStr, translationID, err)
	}

We're throwing the warning even when the error is nil. That block should be wrapped with if err != nil.

As for why it's choosing the wrong i18n file, I am suspicious of how the files are ordered. I'm guessing the logic is something like, "Yeah, zh-CN looks good, let's use that. No need to look any further."