I think your feelings and fears are valid, but ultimately, IMO you should make the decision of whether or not to have a family, and then make a plan for dealing with the anxiety. Because, it can
be dealt with. You can
manage your anxiety while pregnant and parenting, as long as you have a plan.
I always had some underlying anxiety that I thought was just "me," but after my first baby, I had a lot of trouble. That was five years ago, and according to my husband, I had never bounced back from it. Then I had my second baby last year, and I really, really struggled through the pregnancy. I was just not myself at all, and it was not good for me or my family. I finally begged for help before leaving the hospital, and it has changed my life. Seriously, I'm not being dramatic. I started taking meds and seeing a therapist, and I can finally cope with life and with motherhood. I don't feel as panicky or desperate about everything like I used to, and my husband agrees that I'm more myself now.
Meds aren't the answer for everyone, but if you're open to finding a solution that works for you, you might feel more confident about your family planning choices.