If it doesn't feel right it's time to reassess. I think if you "sometimes agree" then you probably know the answer.

Best of luck with your decision