Before getting married, my spouse and I agreed that neither of us cared to know of the others' possible dalliances... and also agreed that such extra-marital affairs were, by definition, permitted.
I wouldn't accuse you of cheating, then. Like you say, the word only refers to things that are outside the boundaries of the relationship (either well-defined or implicit, as the case may be).