I think there is a difference to be made between the physical sex and the genetical sex. Genetically, the sex is determined at the fertilisation of the ovum ( always X ) by the sperm ( X or Y, thus the gender resulting of the father ).
BUT! Physically, the determination is not made until later during the pregnancy and the difference of genitalia results of the hormones ratio created by the fœtus/placenta/ even mother. That said, these facts are true;
- The genetical sex IS determined by the father, at the moment of the conception.
- All fœtuses are PHYSICALLY no-gender (what some may call « female due to the absence of penis) until later in the pregnancy, when hormones ( testosterone mostly ) will make the organs grow into their final aspect, thus defining a physical boy or girl.
The Hermaphrodite cases are probably the result of a lack or abundance of hormones, making the physical and genetical sex different.