You have to set up boundaries, limits, and live your own life. Be blunt, tell him quietly away from the kids that he is calling too much and coming over too much. And emphasize that after all that IS why you are divorced. You need your own space.
He needs to seek help for his problems, you can't do it for him. Tell him he needs a professional to talk to. And get someone for yourself to help you deal with him, as I am sure you need advice and support after 20 yrs of that.
Being peaceable and presenting a united front is great for the kids. He's taking advantage of you by using that, "for the kids" to discuss his personal issues. You gotta sever it.
What I think, anyway. Hope it helps.
"Do something everyday to make tomorrow better."
"One good change creates another."