Yes, the Port Authority: aka the world's worst bus station. But you go a few blocks and you're in Hell's Kitchen which I find a pretty interesting neighborhood with nice ethnic restaurants. Or walk south 10 blocks or so into Chelsea.

When I'm in NYC I often stay at about 42nd and 10th out of choice even if I'm not at a Javits event. It's near things but is out of the craziness of Times Square itself.