I have trouble with it too. I grew up in the northern US and there, we're a LOT more laid back about it then those here in the south. We're not made to address people as 'sir' and 'ma'am'. Down in the south, they flip out if you don't say it. Accuse you of being disrespectful and flame you. Sorry, but I'm a proud yankee even if I do have to live in the south. I got in SO much trouble my first day of school down in the south. I tried to explain that I was from the north and we're not taught to say it and it isn't disrespectful to not say but they didn't care.
To be honest, I hardly ever say it. I say it when I'm on the phone with certain people but I've been raised to believe it is not necessary to say it. Saying something in a polite tone is just as respectful. No need for the titles, imho.
My fiance, every now and then, when he starts to get in the role of being Dom will have me address him as sir but 9/10 times, he hates to be called sir - makes him feel old.