It's a tough call.
I guess the only real answer is, if you think there's something there, you should go for it and talk to him.
If you really like him, then you can't just "turn it off" because you're friends with his sister - and you might be missing out on a chance to be really happy if you do.
BUT.... You just have to proceed with caution and be considerate to your friend. She might find it weird, things might even get awkward between you, but if you're good friends, you'll work it out.
I remember when I was in highschool, one of my best friends had a sister in the grade below us. He used to get really angry and defensive if anyone commented on his sister's looks etc... Probably because he felt the need to protect her and look after her.
Your friend might feel the same way, although not everyone does.
Some people are totally cool hanging out with their sibling's boyfriends/girlfriends, other people are kind of 'weirded out' by the idea.
Your friend might also be worried that there are things that she'd previously shared with you in confidence, in the way that friends do - and if you get closer to her brother, she may not feel as comfortable sharing personal matters in case he finds out.
All of this might not be an issue, but worth thinking about anyway.
