First, it doesn't matter what others thinks, you keep to your feeling & enjoy your relation. Secondly, it's a great moment anyway, "Fake friends believe in rumors, true friends believe in you" this feeling is something between you & him, if a relation suppose to be destroyed by a simple rumor, its better to be ended as soon as possible. He won't get any wrong idea cause he need to believe in what "you" saying to him directly, if he don't, or get doubts or get impressed by the rumors, then he's not a promising person nor a suit person for you. The first stand of a relation is "Trust" if it don't be there, a relation can stand strong & won't work.
You guys already talked to each other, you know his feelings, he knows your feelings, if you guys didn't spent enough long time together, talk to him about it one more time, then wait for the result, if he believed you, means things are serious, if he didn't believed & stood by the rumors then leave him right away.
