One way to possibly build trust (and, as above, I am looking at this without necessarily all the facts) may be to not only talk about it with your sub (because it does show trust) but to also, maybe, look at it in a lighthearted way. You could, for example, mention the thiings he asks to do for you and maybe build self esteem by pointing out that you thought he could not do that as well as your sub could.

I also think a frank and open conversation with this guy in the office may also be warranted, not anything official but maybe a word about 'inappropriate office behaviour' and the need to keep a distance professionally.