I agree with what you're getting at, but at the same time it's reasonable to say that a non-trivial number of people are offended by the term.. So even while most people don't take offense, I don't think it's wrong to describe the word as offensive.