I think it's the lack of rivalry that hurts our interest, and the lack of tradition on top of that. Competitive sports are more entertaining when there are grudges involved... because we all didn't grow up with grudges against any particular soccer team, we don't have a reason to watch the sport.
However, baseball? Yes. American football? Oh yes.
If you think about it too, European colonization probably introduced soccer to a large part of the world in the mid 1800's around the time when the United States was already well established. True, American Football did not really start to take its modern form until the early 20th century, but the modern version of Football itself has only 50 years on the American game.
It's not that we're too stubborn to give in to the rest of the world. Soccer has just had a much wider representation through some massive colonization by the European countries in the 1800s.
Probably a lot of the resentment you see in the so-called "haters" of the sport are due to the fact that there is plenty of arrogance from the rest of the world that soccer is the "world's sport", where it is then insinuated that what we play is some sort of illegitimate, mutated version of soccer/rugby, compared to what we should ACTUALLY be fans of.
We here in the US are no more disrespectful to soccer than European fans are towards our football. I've seen a lot of mudslinging coming from their side in online discussions about this very topic, but what I always ask is, "Why don't you play American football?" ... and the responses are always remarkably similar to our responses to soccer. "I don't understand it/I didn't grow up with it/I don't care about it".