Some do. Others care about the girls brains.
As human beings, we unconsciously look at every part of the people we're attracted to, even the facial features that nobody would notice. Yes, guys do sort of care about the size. We might even wish our partner's were either bigger or smaller. Either way, we still accept them because it doesn't change who they are.