Build XML Sitemaps with indexables

[diedexx]

Based on #17757
Replaces #17580

Context

• Our indexable model is better suited for sitemap generation than what WP Core provides. It can be more performant and easier to follow.

Summary

This PR can be summarized in the following changelog entry:

• XML sitemaps are now being fed from the Indexables database.

Relevant technical choices:

• Decided to remove the wpseo_sitemap_exclude_empty_terms and wpseo_sitemap_exclude_empty_terms_taxonomy filters. To remove terms from the XML sitemap, make sure they're set to noindex in Indexables.
• Removed the wpseo_hit_sitemap_index action.
• Removed the transient caching (which we defaulted not to use anyway so nobody had on probably).
• The wpseo_sitemap_entry filter has the object_id, so the WordPress ID for the object, as its third parameter now.

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

Prep

• Keep two sites side by side, one with the Yoast SEO 17.8 or trunk and one with this branch.
• It's easiest to let the settings/posttypes/terms/content of both sites match as closely as possible.
• Make sure that the site is fully indexed with either the Optimize your site button or the wp yoast index WP-CLI command.

Sitemap contents

• Visit the sitemaps on both sites and compare them.
• They should have a sitemap index entry for all post types that have posts that can be viewed from the frontend and don't have robots:noindex.
• When you click one of the post-type sitemaps, the archive of that sitemap should be in the list on the first page of the posttype sitemap. If the archive is disabled, it shouldn't be in the sitemap.
• The same goes for taxonomies, with the exception of the archive as taxonomies themselves don't have an archives (terms do).
• There's also a sitemap link for the page sitemap.
• Click it and see that all publicly viewable pages are in that sitemap as long as they can be viewed and don't have robots:noindex.
• The first page sitemap should also list the homepage of the site.

Pagination

• add the following line of code to the wp-config.php add_filter( 'Yoast\WP\SEO\xml_sitemaps_entries_per_sitemap', function($entries){return 5;});
• Verify that every page of the sitemap shows no more than 5 links, excluding the post type archives and home page links on page 1 of the cpt and pages sitemaps.

Last modified dates

• Verify that the last modified date are all in UTC.
  • Change the site time to something other than UTC to confirm this (wp-admin->settings->general)
• Verify that the last modified date on the sitemap index all have the value of the most recent last modified date of its children. ignoring post type archives and the home page on page 1.

Excluding objects from the sitemap

• Play around with visibility of posts/pages/terms. The following actions should prevent a link from showing up in the sitemap and should change the last modified date of the parent link in the sitemap index accordingly:
  • making posts private
  • making posts password protected
  • setting the post or archive to noindex
  • changing Yoast SEO search appearance settings to noindex a whole posttype or archive with the "Show X in search results" setting.
  • Create a posttype that isn't publicly viewable, using a plugin like Custom Post Type UI (note that changing the posttype visibility doesn't work without a full indexable reindex as this is not an expected usecase. instead create a new posttype)

Images

• Add an image to a post
• See that the image counter on the posts sitemap has increased for that post.

Test instructions for QA when the code is in the RC

• QA should use the same steps as above.

Impact check

This PR affects the following parts of the plugin, which may require extra testing:

• This has a HUGE impact on XML sitemaps of free and other addons. Please test thoroughly.

UI changes

• This PR changes the UI in the plugin. I have added the 'UI change' label to this PR.

Other environments

• This PR also affects other environments and needs to be tested there.

Documentation

• I have written documentation for this change.

Quality assurance

• I have tested this code to the best of my abilities
• I have added unittests to verify the code works as intended
• If any part of the code is behind a feature flag, my test instructions also cover cases where the feature flag is switched off.

[diedexx]

replaced by feature/indexable-sitemaps