If you have a strong family/ or network of friends, your child will probably not suffer. Yet, you put a lot on yourself by being the only parent. Raising a child is rough. When you go it alone, you don't have someone there all the time to rely on if you need a break, need support, even someone to bounce ideas off of concerning punishments, schooling, etc.
In my opinion it would no doubt affect your child. He will notice that every friend of his is having a dad, while he hasn't. So, at least there has to be a man who takes care of him and express love to him such as your brother or someone who's delighted to do so. Even that if your child has no idea of that matter, he will have the feeling that something is missing and needs to be situated at home, and there has to be a day that will come in the future of the child, and then he would find out at once.
Your child should be brought up fine but the thing is, they will always feel like they are missing a small part of them, especially by the time when they realise that dad isnt around. I know somebody who doesnt know there dad, they were brought up just fine, he has two brothers aswell. Everything will be ok. =]