Does Mastodon have something to the effect of robots.txt (or is robots.txt already robust enough to express an answer to the question "This site administrator is willing to have these toots indexed for search?")
At the end of the day, it's a federation protocol. The protocol's creator can have an opinion on whether things should be indexed for search, but it's up to users of the protocol, not the protocol's creator, how it's used.
(... and I have to comment on this bit from the article. "The problem would be a public search engine that Gamergaters and Kiwifarmers use to hunt down vulnerable targets." This heavily implies that Gamergaters and Kiwifarmers can't just build that for their own purposes, which would, of course, ignore robots.txt settings).
We're already seeing massive searchability problems from the mass migration of communities to discord servers, it's not a good thing and doesn't make anyone safer.
The choice of bogeymen makes me suspect the motivation is insincere, though.
It's not insincere. Many of the early adopters were social (queer, trans, furry) or political minorities (socialists, anarchists). They adopted Mastodon because they were sick of being targeted on Twitter and other centralized platforms. I can understand very much why they are protective of their space.
Yeah there are vanishingly few online queer spaces not on Discord for pretty much this reason. The ability for server admins to bring down the banhammer and actually moderate bad behavior rather than leaving it to central platform moderators that do nothing about all but the most blatant harassment.
Semi-public semi-searchable discourse is really quite nice when your aim is to not have the spotlight on you. Because there are unfortunately people that actively search for people to target and being on Discord makes that in practice impossible.
> Does Mastodon have something to the effect of robots.txt
Mastodon has a "Opt-out of search engine indexing: Affects your public profile and post pages" setting, which probably inserts a "robots" meta tag. [1]
But this is just for ordinary web crawling, and the bit isn't sent along with your posts that get federated to other instances.
Another thing -- noindex is not federated. I used to enable noindex, and still found my content on Google (mirrored on other instances that I was federated with). I just allowed indexing.
Hence the crux of the issue. You can opt-out of indexing. The moment you federate, and your post leaves your server you no longer have that guarantee. All it takes is one instance to boost your post to someone who enables indexing.
True. I was still speaking of the polite space and what the protocol can offer.
In the impolite space, these problems are addressed by encryption and auth/auth, not polite protocol. Which defeats much of Mastodon's designed goal IIUC.
At the end of the day, it's a federation protocol. The protocol's creator can have an opinion on whether things should be indexed for search, but it's up to users of the protocol, not the protocol's creator, how it's used.
(... and I have to comment on this bit from the article. "The problem would be a public search engine that Gamergaters and Kiwifarmers use to hunt down vulnerable targets." This heavily implies that Gamergaters and Kiwifarmers can't just build that for their own purposes, which would, of course, ignore robots.txt settings).