I have the same problem w/ drinking. I ENJOY it fully while I am doing it, then the next day, I start to get fearful. I wonder if I embarrassed myself or if I was rude or said or did something inappropriate.
I started to think about it and sometimes (I only drink on Sat nights, I don't drink during the work week) it isn't worth drinking. However, if during that time I feel really good, i don't want to deprive myself of that either.
My anxiety after drinking is relatively new. I am 32 years old and have drank my whole life and I always had a blast and never was worried the next day. I find now, that if I drink too much and can't piece every detail together, that is when my worries start. I am in a relationship of 13 years and I have never cheated, but I worry after a night of drinking that I have slept w/ other people. I also have OCD, so that is one of my obsessions I worry about. It is awful. It's only been like this the last year or so. My fears are unfounded, but nonetheless, they are scary to me.
So, sorry to be ambiguous on my answer. I think if you feel good drinking, you shouldn't give it up. If you are getting worried the next day and it's not worth it, then don't drink. Feeling good is what counts. I am not on meds, so I don't have to worry about mixing alcohol w/ meds.
Good luck with everything. Panic is a very hard thing to live with. I've had it for 7 years and it sux!