Am I being that selfish?
Not at all. Why should you have to settle and risk tremendous unhappiness? You want what you want, and there's nothing wrong with that. If it is truly a need that he can't satisfy, where do you see the relationship in 5 years or 10 years? Life is too short to waste it being unhappy.