It appears as if he is saying that what you said is ruining the friendship between you and that guy. At least that's what I'm getting. If he gives you an example of when he felt the way you do but held back from it, he is most likely trying to say that you should've done the same about your feelings.
Don't be discouraged when I say this. If a guy ever says something like that to you instead of gently turning you down, you should just forget about him. He's not that mature and doesn't have that much finesse for saying that. What he said can actually be considered somewhat rude. It's as if he doesn't respect your feelings and doesn't want to acknowledge them.
Anyway, I wish you best of luck.