Yes, absolutely yes! I would live anywhere my love was. I could care less where we are, as long as we're together!
Would you live somewhere you absolutely don't belong to live with the person you love? He is attached there because he has responsibilities.
This is strictly your choice but from experience if you hate it there youll be miserable and that will soon take its toll on your relationship.
So now you must figure out if this person is worth living miserably with or move on and be happy yourself somewhere else. Good luck