You need a good random sample and lots of manpower to manually label them. Mechanical Turk [0] is one place to go for that man power if you don't have a "grunt work" team and are not willing to spend a few days doing it yourself.

There are also some methodologies out there that can help you label data sets more efficiently. I don't often see them used, but they exist. Look up "active learning" and "semi-supervised learning".

[0]: http://nlp.cs.illinois.edu/HockenmaierGroup/Papers/AMT2010/W...