That said, your boyfriend isn't with her, he's with you despite the fact that he and the other woman have a child together. On some level, your boyfriend and his ex have already determined that they're not right for each other. There might be some residual "what if?" feelings between them, but such feelings are fairly natural and aren't necessarily an indicator of genuine want.
Even though you can't relate to his exact situation, you can find a way to empathise with him on some level. Maybe you've wondered about how things would be different had you remained with one of your exes or pursued a past crush. If so, then you'll understand that wondering and wanting are two different matters, and the existence of one doesn't necessarily imply the presence of another.
If your concerns over your boyfriend's feelings toward his baby's mother are more than passing and are putting pressure on your relationship - not to mention yourself - then communication is the key to resolution. Enter into a discussion of your relationship and what his fatherhood means for your present and future. Share your thoughts and concerns with each other. From there, you can determine what you believe and what, if anything, you want to do next.
Most importantly, you should try to keep an open mind. Your boyfriend is in a difficult situation and it's only natural for his mind to wander to the mother of his child every now and again. In the end though, it's you who he's chosen to be with. That means something, and if there's love in your lives then there's room for you to discuss this together.