You say that he rarely goes out with his friends? You should be thankful for that alone. Perhaps you've just been around each other too much and when he's gone, these feelings arise. If he hasn't done anything to make you jealous and cares only for you then you need to relax and know that he's enjoying himself. Replace the jealousy with happiness and you'll be better off. Everyone needs some time away. Just let those feelings go and enjoy your time together. Jealousy isn't good for you. Be happy !
