If I told them they didn't really know how to use a browser because... they didn't know to use tab to move between fields... I wouldn't be surprised in the slightest if they came back with some comment about the subtle elitism
Ha, fair enough. But that really only shows that my example wasn't great.
If you eat soup with a fork, you don't know how to use silverware. If you drive nails with a ball-peen hammer, you don't know how to use hammers. That's not hammer or silverware elitism.
IMO, using a mouse in vim shows that you don't know vim. You can still get the job done, but why not commit to using the tool as intended?