I'm not claiming that the system we have is a whole lot better, believe me. But if you had a system where 55% of the people could make it illegal to be a member of the other 45%, would that be any better? What you're saying is that a voting majority could theoretically pass a law to put an unjustly hated minority into concentration camps, or extermination camps, and it would be justified! Sorry, I don't think so!
Personally, I think the world would be better off if we didn't need governments. But given human nature I know that's a pipe dream. But rule by mob isn't much better than anarchy.