I don't know if I am going to be correct with this statement but I will give my two cents on this.
Even though people are into the "lifestyle" they must still live vanilla. They shop, work and interact with others on a daily basis.
Like gagged_Louise said, you tend to detach yourself from others because they just do not share the same things you do anymore.
People in this "lifestyle" will gravitate towards people who are involved as well. They share a common bond and understand each other better.
That is not to say that your vanilla friend you had since grade school cannot be in your life.
But as life goes on, everyone changes and evolves, as do friendships and relationships.
I think what people are trying to say is that even though they interact with vanillas on a daily basis, they are more inclined to make friends with people who share their interests in this lifestyle.
They more than likely do have friendships and interact with vanilla people, but in the long run, it is easier to keep and maintain a friendship with someone who shares a common bond.