I mean, a bad hire is going to cost me thousands of dollars - if not tens of thousands of dollars. $500 is pretty cheap to figure that out up front on a candidate I am otherwise ready to hire. I am not going to do this to 100 people and pick the best, I am going to do it for someone that in every other way is a good to hire dev.

I don't think I would give out the same work to multiple people, let them put it into production on their first day and get the pride going ;)