A harem? Pffft! I want a cult that worships the singular wisdom of the great Wiscoman. Church services are daily, ladies only

Seriously though, you've got to make sure that a) everyone's OK with that and b) everyone involved knows about it. If you're getting a little on the side, then you're a cheater -- D/s relationship or vanilla. Trust means everything (OK, almost everything) is right out in the open.