I want to leave him but I know things aren't gonna work out but I guess I still love him and I don't want to be alone.
You want to leave. You KNOW things aren't going to work out.
Time to tell him, in no uncertain terms. Time to develop an exit strategy and get used to the idea of raising your children alone. Time to see if he can/will rise to the challenge.
If his birth family comes first, it's up to him to change his priority - assuming he wants to. I've been there/done that, it's hard! You can choose to stay and play 2nd or 3rd fiddle, with your children even further down the list, or not. Up to you.
Try to see yourself as the beautiful person you are - his actions do Not define you. Better to be alone, even if you struggle financially, that to be ignored and disrespected. Your kids need you to put yourself, and them, first - they need to see you strong and determined.
Take care ... let us hear from you.