Sitemap generation includes CMS StructrualPages, which it should not

[stevejalim]

Description

In the CMS we use StructuralPage create branches/folders in the page tree. These StructuralPages do not have content but allow us to structure URL paths as we need, including matching some routes that already exist in the page tree as static/Django-only paths

However, we've spotted a bug: if there's a static path for foo/ and a StructuralPage for foo/, the static path is dropped from the page tree, as can be seen here: mozmeao/www-sitemap-generator@bdd1cc0

Expected result

Paths for StructuralPages should not steamroll over static Django-view paths - only add to them

Actual result

Paths for StructuralPages did steamroll over static Django-view paths - leaving only /en-US/about/ in the sitemap, rather than /<all the other locales>/about/

Note that the /about/ page is still available in all the locales it was originally -- this is just about sitemap generation

[stevejalim]

The clearest solution, as discussed with @alexgibson, is simply to ensure that StructuralPages are not included in the lookup of pages used for the sitemap - we never want to send anyone to them directly, after all, and visiting a StructuralPage just 302s to the parent path.

Current code:

bedrock/bedrock/sitemaps/utils.py

Lines 191 to 207 in 6fa3b0e

	def get_wagtail_urls():
	urls = defaultdict(list)
	# Get all live, non-private Wagtail pages
	for cms_page in Page.objects.live().public().order_by("path"):
	# We don't want the Wagtail structural Root page, nor the
	# site root page, because that isn't surfaced from the CMS (yet)
	if cms_page.is_root() or cms_page.is_site_root():
	continue
	_url = cms_page.get_url()
	if _url:
	lang_code = cms_page.locale.language_code
	_path = _url.lstrip("/").replace(lang_code, "")
	urls[_path].append(lang_code)
	return urls