I have to say, one, this is something you've "heard" and its something he denies. Two, if it is true, so what?! This is something that happened long before he met you, everyone has a history. I kinda feel this should not be an issue in your relationship, you either except him for who he is, or you don't!