Ignore. If he tries to make you feel anything, ignorance is the best answer, it'll burn him down, plus, your relation is with your best friend, enjoy and don't let anyone stand in between, it's no one's business and this friendship belongs to you and your friend, no one else can decide it.
However, i guess another case! Maybe the older brother thinking you're being a bad influence on your friend or any other misunderstandings, so that's why he showing his displeasure with such behaviors, you can try to earn his trust and communicate. If it worked, fine, if not, never forget what i've said in the first paragraph.
