If you don't know him well ... Chances are he doesn't know you well.
And if he doesn't know you well ... He can only be physically attracted to you.
You have to actually put some effort and time into getting to know somebody, before you know if you like them.