I went for a interview with Facebook recently from India and had a really different experience. The CBP guy was really gentle and just reminded me to not work for Facebook in my days there and that's it, I was in USA. I just had a small carry bag and no luggage so another one of them asked me why I have no luggage and I told them that I am only there for 3 days. I agree US can be a bit paranoid of these things but compared to people trying to enter USA I think very few face the kind of hostility the author faced.
Agents use their read on a person to determine entry/denial. The OP mentioned all the things he was asked and how he answered. He didn't/couldn't explain his attitude, eye contact, how quickly he answered the questions, facial expressions.
You've met people you took an instant dislike to - maybe its their hyper-vigilant appearanch; maybe they involuntarily smile at everything you say making you think they know something you don't know. Things like that can get you turned away at the border.
