If these people don't want people to fantasise about them, that's hard luck and I can't feel sorry for them. I can't stop people fantasising about me any more than they can stop me about them.
Certainly you have a point. However, I still think there is a big difference between fantasizing about someone and using their name in a published story.

Bill