I will use it in one instance...
And that is the one instance. Almost 100% of the time, I'm using it to inwardly call someone a dumbass.
Yes, very true and I do love it when that happens. Unfortunately, the majority of the ones that bug me are not that. Between the election coverage and a few podcasts I've listened to recently, the interviewee says it before they answer every question. It's like they stupidly said it for the first question and are now afraid that if they don't say it after every question, that the interviewer will somehow think the most recent question wasn't a good one.
I've actually gotten pretty bad with people, at least in a social situation with the "At the end of the day." As soon as someone says, "At the end of the day." before they can say anything else, I just say, "Yeah, at the end of the day, it's just the end of the day." I get some odd looks, but at least it makes me feel better. My wife gets really pissed at me for it though.