Hi BK - you are a wonderful, intelligent, strong human being who is being used by some members of your family. They are the ones with the problems and not you. You have your own family now who love you and want you to stay healthy for them. Distance yourself from this man and his mother. I am assuming that the grandma you refer to is his mother.
Spend all your energies on you and your family. Go out and do things as a family and don't ever put yourself down because they have you believing you are the problem. Your mom sees whats going on and maybe she will guide you from heaven. Just don't sell yourself short. Raise your family - be happy and forget all the negative stuff they throw out there.
You can do it.