It is a civil rights issue, there's an equal protection argument to be made. Further, there's an unenumerated rights argument.
As for "the people", it simply shows the tyranny of the majority; something our government was specifically designed to guard against. "The people" might speak, but they don't get to restrict the rights of others no matter how loudly they do so.