If he wants to be more sexually active than what is happening right now, and talking about it doesn't help, then yes.
Because of your dissatisfaction, you are now looking at other guys, perhaps thinking that it could be different with someone else.
When you are in a relationship with someone else, your goals need to be similar or you end up fighting for what you want. I would not want to spend my time fighting with someone. It's really not pleasant.
So if you two cannot get on the same page sexually and he keeps pushing your boundaries, then yes, the best thing would be to break up.