Some people will say yes, I however feel that lust is more the definition that you're looking for. To say you love your wife is not so, or you wouldn't have a mistress, to say you love your mistress is not so or you would be divorcing or divorced from your wife. To me, you are taking the informed choice away from your wife who I'm assuming does not know of the mistress, and the same is true of the mistress if she is not aware that you are married.