I think why you feel this way is because many people still see (forgive me but I'm an "old fashioner in this way) sex is the ultimate intimacy that generally comes with feelings of love.
Unfortunately for men and women (sorry women can be heinous users of the opposite sex, so can men) alike it's not always the case. Maybe some of us are still "wired" so to speak in the same manner as our ancestors, others of us believe in the pack mentality. No I'm not talking about an orgy in that case, more like the fact many mammals have one mate for life who helps to lead the pack or at least shares in it.
Casual sex and the emergence of it kind of changed the idea that you give your sex to the one you love if and only if they love you in return. (Yes my mom gave me THAT talk). I don't think you need to change the way you think. Sex with love is...absolutely AMAZING, but you also need to identify if you're actually in love or in lust with the person you're with. Communication is important, but one thing I think we forget is that communication with *YOURSELF* is the most necessary.