I am not poly... and don't intend to be.. but I am just gonna throw my two cents in for fun.
If I was to ever consider a poly relationship- I think that I would have to have a relationship with both the man and woman already in it. I'd want to have a personal sexual relationship with each of them.. so it didn't feel like just 1 guy with 2 chicks, ya know? I'd have to have something where each "pair" has their own personal love and relationship between them. In my mind.. that is the only way I could make something work... that... and assuming the original couple made me feel very included and didn't spend a lot of "private time" together... especially at the beginning.