14.x compatibility - Wrong indexation data with language plugin

[Djennez]

This issue is written with WPML as example. However, I believe this can happen with more language plugins.

---

Edit:
The WPML - Yoast SEO add-on, obtainable via your WPML account, prevents this issue from happening.

To reproduce:

1. Install WPML and Yoast SEO, and not the add-on
2. Configure WPML to use 2 additional languages during its setup process
3. Create 3 posts in the 3 different languages and add every posts language to its own version of the uncategorized category (for simplicity's sake)
4. Switch to a language from WPML via the language selector in the admin bar
5. Reset the indexables via the Yoast test helper
6. Rerun the indexing process from SEO -> Tools
7. Visit the 3 different uncategorized category pages and check their canonicals. They will share the same canonical across all 3 languages and it will be the canonical of the language that was selected before resetting and re-running the index

With the addon, the wrong data will still get saved to the database, but it will not be used for displaying the canonical. I am not sure if this will still cause problems in other places. Breadcrumbs and canonical in the meta seem to be fine with the addon.

---

While performing some additional tests with WPML I found that running our indexation may cause wrong permalinks and other details like breadcrumbs to get saved to our indexables database.

My setup:
WPML with 3 languages: EN (default), NL, DE
Set language URL to: subdirectory, but subdomain has the same issue

WPML by default creates the "Uncategorized" category for every language created, so we can use this for testing.

This issue only seems to happen when the index is run, not from individual created terms / posts. So reset the indexables / migrations and start the index.

If you check the indexables database, you will find the permalinks for the 3 terms for uncategorized to contain the same language identifier:

objects 1, 5 and 6 are all uncategorized but in different languages

This identifier seems to be depending on the language you're currently viewing the site as. So in the example above I had WPML set to Dutch (NL). Switching to German (DE) and resetting the indexables again, I would end up with DE identifiers in the URL. This will cause all sorts of mixed up details in webpages, like this category archive with the wrong canonical / og:url:

In the example below I have set URL structure to use a subdomain as language identifier and I was logged in with WPML set to German when I ran the indexation:

And after resetting the index and saving the terms individually, these are the results in the table:

These are the values I would expect.

[monbauza]

Please inform the customer of conversation # 616185 when this conversation has been closed.

[monbauza]

Please inform the customer of conversation # 623651 when this conversation has been closed.

[monbauza]

@Djennez may we consider giving a higher priority to this issue? Not many users have reported the issue so far but probably they're not even aware. As this issue affects canonicals as well as breadcrumbs and doesn't have an easy workaround, it may have SEO implications on users' sites.

On my test site, the problem occurs even without running the indexation process. See the steps below.

1. Set different languages in directories in WPML, i.e. English (default) and Dutch
2. Enable Yoast breadcrumbs and select to show the category on posts (SEO > Search Appearance > Breadcrumbs)
3. Create a post in English and assign a category to it
4. Translate both the English post and category to Dutch
5. Reset indexables tables
6. Visit the Dutch post in the frontend. Note that the breadcrumbs are correct.
7. Visit the Dutch category, inspect the source code and note that the permalinks are also correct: canonical, og:url, etc.
8. Visit the English post in the frontend. Note that the breadcrumbs include the Dutch category (see image one below)
9. Visit the English category, inspect the source code and note that the canonical URL and og:url are wrong; they point to the Dutch category (see image two below).

[priscillamc]

Please inform the customer of conversation # 624424 when this conversation has been closed.

[Pcosta88]

Please inform the customer of conversation # 640342 when this conversation has been closed.

[monbauza]

Wrong canonical is also outputted on translated custom post type archives. See example below for the Spanish locations archive. If I reset indexables and visit the Spanish archive page first it's the other way around: the correct canonical URL is outputted on the Spanish page but the canonical URL on the English page is wrong (it points to the Spanish version).

WordPress v5.5.1, Yoast SEO v14.9, Local SEO v13.5, WPML v4.4.2 and Yoast SEO Multilingual v1.2.4.

[monbauza]

Please inform the customer of conversation # 650657 when this conversation has been closed.

[Sjardo]

Any fixes for this issue on the roadmap?

Looks like the same issue as #15582.

[Beee4life]

@Djennez have you tried with v13.5 to see if it works there ?

[Djennez]

@Beee4life nope, this issue is with 14+ specifically because of the indexables feature.

[Pcosta88]

Issue happens with TranslatePress v1.8.5 and Yoast SEO v15.2

[Pcosta88]

Please inform the customer of conversation # 662917 when this conversation has been closed.

[monbauza]

The problem with breadcrumbs including the translated category URL still occurs in the latest version of the plugins: Yoast SEO v16.1.1, WPML Multilingual CMS 4.4.10, and WPML SEO v2.0.0. I've managed to reproduce the problem by following these steps:

1. Install and activate Yoast SEO, WPML Multilingual, and WPML SEO
2. Set different languages in directories in WPML, i.e. English (default) and Spanish
3. Enable Yoast breadcrumbs and select to show the category on posts (SEO > Search Appearance > Breadcrumbs)
4. Create a category in English
5. Translate the category into Spanish
6. Create a post in English and assign the category created in step 4) to it
7. Translate the post into Spanish, add the Yoast Breadcrumbs block and publish the post
8. Inspect the Spanish post in the frontend and note that the category item in the breadcrumbs points to the Spanish category
9. Make an edit to the English post
10. Inspect the Spanish post in the frontend again and note that the category item in the breadcrumbs now points to the English category (see screenshot below)

Note: the canonical tag seems to be generated correctly on both the post and the category pages for the two languages

[Pcosta88]

Please inform the customer of conversation # 725330 when this conversation has been closed.

[Pcosta88]

User reports canonicals are different than permalink when using TranslatePress v1.9.7 and Yoast SEO Premium v16.1.

User says that re-saving the post types is resolving the issue.

[monbauza]

Please inform the customer of conversation # 741002 when this conversation has been closed.

[monbauza]

Please inform the customer of conversation # 822982 when this conversation has been closed.

[laurasacco]

Please inform the customer of conversation # 827526 when this conversation has been closed.

[maderafunk]

@Djennez
This issue still exists with Yoast SEO Version 18.5.1 and TranslatePress - Personal Version 1.09 and TranslatePress - Multilingual 2.2.4.

Somehow, some parts of the canonical url are correct, while others are not.

For example:

Correct URL in German:
https://.../de/produktkategorie/damen/kleider/

Canonical URL created by Yoast plugin:
https://.../de/produktkategorie/women/dresses/

For comparison, the English URL, looks like this:
https://.../product-category/women/dresses/

So somehow, product-category has been successfully translated in the Canonical URL to produktkategorie, but everything following the product-category is not being translated anymore.

I have only noticed this behaviour on the Woocommerce pages, all other pages seem to be translated correctly.

How can we fix this, even if it is a temporary fix? Is this a bug from the Yoast Plugin, or from Translatepress? As it also affects WPML, I guess the bug is on the Yoast Plugin side?

This has severe implications on SEO rankings, It would be nice if this could be treated with high priority.

[rumejan]

Please inform the customer of conversation # 885438 when this conversation has been closed.

[suascat]

Please inform the customer of conversation # 885625 when this conversation has been closed.

[maderafunk]

The canonical links were still incorrect for me, so I wanted to deactivate them completely by adding this line to the function.php of my theme:

add_filter( 'wpseo_canonical', '__return_false' );

Surprisingly, this did not deactivate the canonical links, they are still there, but it fixed all the problems! I do not understand why, but suddenly all canonical links in all languages are correct.

Can someone explain this, and could this help to fix this problem once and for all @Djennez ?

[maderafunk]

TranslatePress has issued an update (TranslatePress - Multilingual Version 2.2.8) that finally solves this issue.

[monbauza]

The problem with breadcrumbs including a category in the wrong language still occurs with Yoast SEO 19.4, WPML 4.5.8 and WPML SEO 2.0.1. I've been able to reproduce the problem by following these steps and different WPML configurations, i.e. languages in directories, a different domain per language.

[monbauza]

Please inform the customer of conversation # 917892 when this conversation has been closed.

[mmikhan]

Issue opened internally on PC-364

[kalilfagundes]

Still have the same issue. Translated page always point canonical as its url (translated slug with /en/ etc) instead of default language.

[JoryHogeveen]

Same here, still the same issue. Also happening with Polylang.

See more in depth remarks here: https://wordpress.org/support/topic/breadcrumb-cpt-archive-translation-issue/#post-17393257

[Sophie-2e]

Same here with WPML and Yoast updated to the latest versions: #20902

[JoryHogeveen]

Lol, is @Yoast that afraid of competition that you need to remove the comment from @Sophie-2e suggesting RankMath?

Anyhow, I do not consider switching plugins an actual solution. That won't fix this bug in Yoast SEO.

@josevarghese
Closing #20902 is fine of course as I can understand that duplicate issues are confusing, though since the issue is already active since 2020, won't it again be forgotten then?
This older issue is tagged for v14 and as a minor issue so my impression is that this will be stalled even more by closing this topic as well.

In case you have some pointers / filters for mee to check I might give it a shot myself. However, What are the chances that such a PR will be accepted (since I'm not part of Yoast).

[josevarghese]

Hi @JoryHogeveen

First, I can understand your difficulties and frustration when the issue takes longer to get fixed. I'm really sorry for taking more time on this.

As you know, for every product, the product team assesses the severity of the problems in relation to other open bug reports and new features. Based on their assessment, the bug report will be prioritized. Our developers also work on the highest priority issues first as they need to get fixed faster and introduce more features.
The good news that I can share with you is the issue #19739, which is within the next sprint of our development team, as they prioritized it after the internal discussion. If this issue is related to it, it will also get fixed on that sprint itself.

Regarding the comment deletion, kindly note that as per the Code of conduct, the respective user commented the same comment on about 3+ issues which is inappropriate and duplication.

We always welcome PR from everyone who would like to contribute to our plugin to make it more awesome. Remember, every contribution, no matter how small, plays a significant role in improving our plugin for users worldwide. So, dive in with confidence, knowing your efforts are valued and appreciated if they can be merged into our repo. You can see the props in our changelog where all those PRs are from outside Yoast merged after thorough testing.

We're eagerly waiting for your PR.

[JoryHogeveen]

Hi @josevarghese

No worries, it wasn't frustration :) And I fully understand that high priority issues will be fixed sooner than low priority issues.

It was just that I'd like to have a bit more insights into the current status and process for this issue, because this is apparently such an old issue.
You just provided that insight, so thank you.

#19739 looks like it's exactly the same issue as this one (and the issue I have when using Polylang).
I've subscribed to the issue so I assume I will be notified once the sprint is over.
In case this issue is still there after the sprint you mention I'll take a look myself.

PS: Regarding the comment deletion, check, that makes sense 👍

[jonasm82]

Hi @josevarghese

The issue remains unresolved and I don't understand why it has been tagged as a minor severity issue? I have a client with multiple multilingual sites using TranslatePress and Yoast and in scans I picked up over a hundred pages with incorrect canonical URLs being set by Yoast which could not be fixed by resetting indexables tables & migrations with Yoast Test Helper. Given that canonical URLs impact indexation, I would always consider this type of issue to be severe instead of minor and resolved by an SEO plugin like Yoast as a matter of priority instead of taking years to attend to? If this is common across multilingual sites using Yoast then there could potentially be an immense amount of sites and pages experiencing indexation issues as a result. Or am I missing something here?

Thanks to @maderafunk - the add_filter( 'wpseo_canonical', '__return_false' ); filter turns out to be a temporary fix for us as well that resolves the issue once applied, although like you I am also not sure why it does. Without it the issue was still present, even with the most recent versions of Yoast and TranslatePress installed.

[jonasm82]

On further investigation it is not just the canonicals where Yoast inserts the wrong URL. It also does it for og:url and application/ld+json markup. So this issue is even further reaching and severe in my opinion.

[julienalbt]

I have the exactly same issue when wanted to display, in a post, related posts based on the Primary category with _yoast_wpseo_primary_category key.
For example, a post with multiple categories, in different language, with the same categories and Primary category across language.
When saving a post in other language than the main language, the _yoast_wpseo_primary_category ID (the translated Primary term id) is replaced by the term ID of the main language term.
So when we want to display related articles with the same Primary category, it won't work because, in the code below, _yoast_wpseo_primary_category will not be the ID of the translated term but the ID of the main language term.
And there is no translated articles with term in main language of course.

$args = array(
          'posts_per_page' => 3,
          'order' => 'DESC',
          'post__not_in'   => array($post_id),
          'meta_query' => array(
            array(
              'key' => '_yoast_wpseo_primary_category',
              'value' => $primary_category_id,
            ),
          ),
        );
        $last_posts = new WP_Query($args);