I myself am in your situation and no, they don't want to leave and they never will but you need to stand up to him and tell him to go I personally have just done the same thing and let me tell I feel better and whether he will admit it or not so does he.