I think you need to sit down with your man and have a real heart to heart with him. In my opinion, there are two ways this could go and one isn't going to be a problem...
A lot of people have fantasies, but that doesn't necessarily mean that they want to act upon them. Some fantasies are better off remaining just that, as the reality is not as good.
For example, I have a friend who has a fantasy about being watched while having sex. She tells me that even with a willing partner, she will not do this as she thinks it would ruin a relationship. Also she thinks it will ruin the fantasy for her if it doesn't go as well as it does in her head (if that makes sense).
However, if your boyfriend is secretly craving adventurous women and multiple partners then yes, you really do have a problem. If your boyfriend asks you to join him, I honestly think you will regret it... but this is a decision only you can make.
As someone else said, this is not loving sex in a one to one relationship, this would be sex for the sake of sex, "dirty sex" if you like. I think you would begin to resent yourself for it. The other thing to think about is nasty little viruses that he (and possibly you) could pick up from engaging in casual relationships with multiple partners.
Whatever either of you decide to do, make sure protection is always used.
Hopefully this is just a fantasy that he gets off on, that in itself is not really a problem (unless you see it as one of course). However, if he decides to explore this fantasy in real life, it might be time for you to find someone who will be true to you.
