I'm a firm believer in going with your gut instinct. If he makes you uncomfortable you need to very clearly tell him that you're not interested. You don't need to be rude or anything, just firm and clear.

I often find it hard to say things that I think will hurt someone's feelings but your safety comes first.