This is a tough and personal decision!
I can get why your P-doc might want to get you a bit more stable...with more recovery time from panic...
...but I also understand you not wanting to wait...because as you said, you just don't know when that might be, and you are ready to grow your family at this time.
I had both my babies pre-anxiety...so I don't have personal advice to offer...but can say that I have read many accounts of moms whose A/P was very managable sans Rx during pregnancy.
I wish you much ease in making your decision...and of course support you in any direction you decide to ultimately go.
"Courage is not the abscence of fear, it is feeling afraid and doing it anyway!"
"Worrying does not empty tomorrow of its troubles...it empties today of its strength."
Corrie Ten Boom
Panic Syndrome recovery due to CBT