This exactly. I've found out long after meeting some of my friends and acquaintances that they were vegetarian or vegan only when it was relevant (meal plans, cateringg, etc.) and never would have guessed were it not for, in other cases, omnivores berating them for their preferences. As someone who is sometimes a vegan, I've actively tried to hide it in public because it's almost never looked upon favorably by omnivores, who as far as I can tell seem to assume that vegans think they are superior to everyone else. Never met a vegan who was this way, though I'm sure they exist.
It probably comes down to the introduction of ethics into the mix. If ethics were not introduced as part of the argument for the dietary choice, most omnivores who make it an issue probably wouldn't.
Personally, I don't care what others eat. They could eat tasteless fiber, or artificial meat-tasting protein or actual meats. It's their choice.
On the other hand, if I'm a guest and there are only vegan choices, then that what there is, likewise if I'm a guest and there is only steak, that's what there is (and I don't like steak).
