Some of this might just be problems with the websites themselves. With the Path Of Exile 2 Wiki example, I'm looking at the sitemaps. Once you get through the first sitemap of nested sitemaps (ugh, not a fan of those), the very first actual sitemap returns:
XML Parsing Error: no root element found
Location: https://www.poe2wiki.net/sitemap/sitemap-poe_wiki-poe2_wiki--NS_0-0.xml
Line Number 18040, Column 1
(EDIT: Of course, after I post this, it's now working again...)
On to the Hytale Wiki example. I don't know if what I'm about to say applies to Google, but I'm approaching it from the perspective of my tiny dumb indieweb indexer for my personal search engine. It is much easier for me to index from a sitemap rather than try to crawl a website, so I basically look exclusively at sitemaps.
Looking at Hytale Wiki, my process:
* Site has a robots.txt file - good!
* Robots.txt mentions a sitemap - excellent!
* The sitemap is stored at /images/sitemaps/index.xml ... oh. I would usually not index anything from a /images/ folder, because I want to index pages only, not images. This would likely trip my exclusion filters. Let's ignore that and continue.
* The sitemap is a list of an additional 26 nested Gzipped sitemaps. Oh.
This is the point where my tiny dumb indexer would stop. Gzipped sitemaps are part of the standard, but they're relatively rare on the web for small sites. They typically only get used if a sitemap file exceeds the 50,000 pages-per-sitemap limit. In this case, 26 * 50,000 makes the HyTale Wiki look like a 1.3 Million page site. Do I really want to index 1.3 Million pages, an estimated 13GB of indexed text data, about a video game I'll probably never play?
My search index is storage constrained, and my indexer is very time constrained. The time I spend indexing your site is time not spent indexing another, possibly higher quality website. So at this point, I'd just grab the front page and disappear... like Google apparently does too.
Of course, the HyTale site isn't 1.3 Million pages, it's only 4,277 articles. That would all fit in the root sitemap file, and that might be the better approach for getting indexed.
Me again! I couldn't figure out why I was getting transient errors on that Poe2wiki sitemap, so I tried looking at other sitemaps on the site. I'm getting transient errors on all of them.
Then I realized - the sitemap files are being overwritten in real time. Every edit on the Wiki is causing the sitemap file to be edited in real time. That's why the sitemaps sometimes stop right in the middle of a filename when I access it - the sitemap file is in the process of being rewritten.
That's a behavior unique to a Wiki, and might explain the entire phenomenon.
Makes sense. Whatever generates those site maps should use the classic Unix trick of writing a separate file then renaming. Also maybe don’t update lastmods on each individual update, it’s a waste of resources since the granularity of lastmod is usually the day (though you can make it the second if you want).
Though weirdly it makes the subdomain result even more interesting: presumably those wikis are rewriting their sitemaps too, yet Google manages to index them normally
The sitemap issues are definitely worth fixing, but I don't think they explain the most interesting part: the same kind of wiki seems to behave very differently depending on whether it's on a fresh root domain or an established subdomain.
There's no need to speculate on whether the sitemap is received correctly by Google since Google Search Console tells you how many pages it can discover from the sitemap you submitted.
The poe2wiki issue is a little different to what is documented, albeit only for the last few months, previously we certainly had visibility issues.
The recent issues relate to site errors due to aggressive crawling of uncached, server-intensive pages (diffs, etc) hidden behind residential proxies. We were being hit sufficiently hard that the server had stability issues and we were penalised by Google.
The sitemap being hosted at /images/sitemaps/index.xml is a problem. Technically, that sitemap can only include URLs beginning with /images/sitemaps/. They should move that file to the root, like /sitemap_index.xml and reference /sitemap-1.xml, /sitemap-2.xml etc.
I wish. Not only do they sometimes randomly fail, Google search console will not tell your why. I've got a site with a sitemap which passes in every validator I could find, yet Google doesn't read it and GSC just reports "error" without any details. Google is big enough to go "we don't care, it's your problem".
On to the Hytale Wiki example. I don't know if what I'm about to say applies to Google, but I'm approaching it from the perspective of my tiny dumb indieweb indexer for my personal search engine. It is much easier for me to index from a sitemap rather than try to crawl a website, so I basically look exclusively at sitemaps.
Looking at Hytale Wiki, my process:
* Site has a robots.txt file - good!
* Robots.txt mentions a sitemap - excellent!
* The sitemap is stored at /images/sitemaps/index.xml ... oh. I would usually not index anything from a /images/ folder, because I want to index pages only, not images. This would likely trip my exclusion filters. Let's ignore that and continue.
* The sitemap is a list of an additional 26 nested Gzipped sitemaps. Oh.
This is the point where my tiny dumb indexer would stop. Gzipped sitemaps are part of the standard, but they're relatively rare on the web for small sites. They typically only get used if a sitemap file exceeds the 50,000 pages-per-sitemap limit. In this case, 26 * 50,000 makes the HyTale Wiki look like a 1.3 Million page site. Do I really want to index 1.3 Million pages, an estimated 13GB of indexed text data, about a video game I'll probably never play?
My search index is storage constrained, and my indexer is very time constrained. The time I spend indexing your site is time not spent indexing another, possibly higher quality website. So at this point, I'd just grab the front page and disappear... like Google apparently does too.
Of course, the HyTale site isn't 1.3 Million pages, it's only 4,277 articles. That would all fit in the root sitemap file, and that might be the better approach for getting indexed.