Hi there and of course we do not think your crazy, no one is crazy here. I have not had that experience but perhaps someone else will be able to address it.
I think staying in the same apartment is not a good thing. If he broke up with you he should move out as it seems to be his hometown can he not move in with a friend, rent a hotel room or whatever. It is so unfair to subject you to his presence and make you feel isolate. That is mental abuse........... talk to him about leaving.
Do you have a therapist or someone you can talk to that is just for you?Keep posting here as we will support you and we care. Hugs to you.