Like I said in my response to jrockway, downvotes matter, especially in the first few hours of a comment's lifecycle. Those first few hours usually determine whether an interesting discussion comes out of the comment, so if the comment is less visible (due to font color and placement on the comment page) due to a few individuals' knee-jerking, it is unhealthy for the community as a whole.

It's like a baby not getting enough nutrients in the first few months. That will adversely affect its development and have a permanent negative impact on its future health.