It's a piece of bad luck for someone who objects to something in your story to bother and give it an unjustly low score.

For stories that manage more than a score of reviews a mechanical solution to the problem is to simply cut the bottom and top 10% of scores before crunching any numbers. I vaguely remember reading how the averages are calculated and I don't recall any such cropping.

I share peoples hesitance at any form of censorship. I suppose the method I suggested above could effectively remove the persons voice but their opinion would be there if you went looking.

The real solution however does not involve maths or social ethics but us.

If you take the time to read a story and you enjoyed it then take the time to review it. Even a small % of the readers of many stories would make up a reasonable pool for scoring purposes. This is something I was pretty slack on while lurking, but something I have recently attempted to remedy.