Its a good question...though I agree there's no easy answer. Sometimes you might see warning signs...sometimes things might not become apparent till afterwards. Life is like that.
But my view is that when a Dom is aware that a sub or potential sub is in fact exhibiting low self-esteem they are better off staying clear. And I'd suggest that'also in the interests of the other person concerned.
I agree it gets back to the issue about informed consent...about actively and willingly submitting...very much different to simply going along with someone or something because one does not see an alternative or cannot reach out for it. In my experience (limited compared to some) a Dom is going to be better off with someone who has some strength and some confidence - it just becomes more fun to know you can twist and motivate and drive that person in certain directions notwithstanding their own 'vanilla' preoccupations or hang-ups.
My current submissive very much enjoys being my slut and my whore...and these terms are used openly by us both. But it reflects our exploration of her sexuality...something that's been discussed at length. I'd never use such terms in order to humiliate or put her down.
And no, don't think that means I never exploit a situation which might be embarrassing or humiliating! *grin*
OTOH...I had a lover some time back...a nascent sub let's say - someone who was more than ready to experiment and have me dominate' her. Again, a lot to do wiht her 'kinky' dark side and enjoying the freedom subnmission could bring. But I never felt entirely comfortable with going the whole hog and taking her as my sub...of becoming her Dom or Master. I still think I could have collared her with only a little bit of encouragement. I didn't - because I sensed that she was mostly exhibiting an infatuation with me...she would have done almost anything I asked because she simply had no way of ever saying no!
Now, perhaps its just a matter of the order of things - maybe if she'd been my submissive first and infatuated down the track. But there was always that doubt in my mind asto whether she'd one day wake up and regret all that she'd experienced with me.
But, to put one other twist in the tail - the aspect of low self-esteem that most interests me from a D/s viewpoint is *trust*
When someone has low self-esteem it may not be possible completely to overcome that. What can be achieved is to build up trust between two people - to create an environment in which self-confidence might be cultivated. And trust is of course vital to good ongoing D/s relationships. So I can understand where a Dom might feel they were able to take on a sub with low self-esteem but still be non-exploitative and to help that individual to grow and develop trust as a pre-cursor to developing more confidence.
Self-esteem and self-confidence - almost any of us could think of a situation where we may lack confidence and might want to hang on tight to someone we can trust and rely on.