While I do agree that being told by a stranger that he was watching me would freak me out a bit, I do have a couple of questions for you. First off: Have you covered all your windows and insured people (including him) can’t see into your home? If people could see in, consider his comment a good safety tip for you.

Second question for you: Removing how he approached you, is he someone who you would have wanted to be approached by? I would not want to suggest disregarding any internal alarm bells within you. In fact if you truly have them, certainly heed them and be direct with the man (as others above have suggested) and tell him you do not wish further contact. But do ask yourself if you have alarm bells or if you are simply jittery. If you are simply jittery, and not in fact hearing the internal alarms regarding him, then perhaps consider setting a few boundaries where you can feel comfortable and you can get to know him. This might be a good test for him too, as you can then watch to see if he respects your clearly-stated boundary. If he violates this, he violates your trust.