I don't see anything wrong with it. Just because hubby and I are looking for another couple to swing with doesn't mean he'd gay or bi. And I used to be friends with a gay threesome; it didn't mean I was necessarily gay, or lesbian, or bi.
People who stick their noses too far into your business need to be told to get out.