I think the more important question than "when is a fantasy too much for reality" is why do you feel shame at your fantasies? What about them is shameful to you, and why do you accept that that aspect of them is wrong?

You touched on the core issue with your comment about "social programming," and "getting past it." Once you've determined what it is about your fantasies that's making you feel shame, then you can examine your premises and decide whether that shame is a valid feeling or not based on your values. If it isn't, then you'd have to work at changing yourself to reject those premises you've accepted in the past that are leading you to feel shame.

Now, all that being said, is it the shame that attracts you to the fantasy in the first place? Do you enter it knowing that when you're done, you'll feel shame? If so, then that's a different matter entirely. Perhaps once you stopped feeling shame for the fantasy, you'd not find it as enjoyable.