Your question is actually a good illustration of why some are accused of sexism. Did you notice that you used different terms in describing men and women? "Larger women" implies fat, obese, or *ah* big boned. "Shorter" makes a statement about a particular attribute: Height.
Perhaps it's generalisations or unwitting slurs that are considered sexist.
Don't take that as criticism. I was just amused by your use of different language in describing men and women.