I feel it's the role reversal. People are accustomed to male led relationships. Anything other than that is odd or out of the norm, simply because they can't or won't comprehend. It's much easier to follow societal norms.

Try to ignore and not allow other people's reaction to your relationship irritate you (easier said than done). Eventually, the people you're around often will become accustomed to it or at least keep their comments to themselves.

Female submissives catch a bit of flack too. My friends and some of my siblings initially thought I was joking when they found out that Sir dictated what and when I'd do something (though we're very low protocol). Their disbelief came as I'm fairly dominant at work and in other areas of my life; that I would defer to someone else, especially a man was unthinkable.