I don't think I can particularly say I was distressed by my fantasies/desires. I was more upset that I couldn't find someone who felt the same way as I did.
I chalked it up to being a bit abnormal and a minority. I knew that I wasn't completely abnormal, just different. Having no access to the internet, or no way of really knowing where to look for likeminded people didn't help things.
I repressed the idea that I would find someone who felt the same way I did. I really didn't repress my fantasies. I kept them as a way to cope with my vanilla life.