It seems to me that any word can be made derogatory. It's the sneer in their tone, in their face, that makes the difference.