Unfit or just bad at parenting?
Shouting at a child, not paying him/her enough attention, leaving him/her with others for long periods of time, not providing stimulation at home and failing to help with school work - these are all examples of bad parenting but they don't make the parent legally unfit. Unfit in the legal sense requires the parent to be behaving in a way that causes endangerment to the child. The specific elements of parental fitness are a matter of state law.
What would cause endangerment to the child?
A mother could be said to be unfit if she was violent towards the child. Obviously you don't want to wait for this to happen so a court will usually consider whether a parent is unfit simply on the grounds that they have a history of domestic violence. If the child is disciplined excessively or is emotionally abused by the mother then she is legally unfit. A history of drug use or alcoholism could be used to describe her as unfit too. Any previous convictions for sexual offenses against children would clearly mean she was unfit.
How to prove it
This depends on the grounds which you are basing your case on. It's easy to prove any previous sexual offenses or drug abuse as it should have been recorded in the court system. This could also be done to show a history of domestic violence.
It's more complicated to find evidence of excessive discipline as what it normal to one person may be abhorrent to another. Emotional abuse is also a difficult one as there is a difference between this and simply being unkind.
Physical abuse is the more obvious. If the child has injuries, document them and photograph them each time they happen. Take him/her to the doctor and ask for professional opinions on how they were caused. If they happen regularly and the doctor says they could have been caused by the mother, present your evidence to the authorities.