I would tell you both to stop the lifestyle for a few weeks. If he is getting frustrated and it is rarely to do with you, then it has something to do with him. As a dom, I know when I should stop and collect myself when mundane life takes premises over my chosen life. And since you have kids I would urge you two to talk about all possible problems before stepping back into the life. It's hard to stay safe when you can't stay sane.