If he's only distracted for a little while, it could be a really nice thing, because it could give you both time to build up some desire. And if you are there for him while he is stressed out, then your connection will be stronger. When you do finally get to have some play time, it will be all the nicer for it.